Abstract. Existing face recognition using deep neural networks is diffi-
cult to know what kind of features are used to discriminate the identities
of face images clearly. To investigate the effective features for face recognition, we propose a novel face recognition method, called a pairwise
relational network (PRN), that obtains local appearance patches around
landmark points on the feature map, and captures the pairwise relation
between a pair of local appearance patches. The PRN is trained to capture unique and discriminative pairwise relations among different identities. Because the existence and meaning of pairwise relations should be
identity dependent, we add a face identity state feature, which obtains
from the long short-term memory (LSTM) units network with the sequential local appearance patches on the feature maps, to the PRN. To
further improve accuracy of face recognition, we combined the global appearance representation with the pairwise relational feature. Experimental results on the LFW show that the PRN using only pairwise relations
achieved 99.65% accuracy and the PRN using both pairwise relations
and face identity state feature achieved 99.76% accuracy. On the YTF,
both the PRN using only pairwise relations and the PRN using pairwise
relations and the face identity state feature achieved the state-of-the-art
(95.7% and 96.3%). The PRN also achieved comparable results to the
state-of-the-art for both face verification and face identification tasks on
the IJB-A, and the state-of-the-art on the IJB-B