OLE: Orthogonal Low-rank Embedding, ´
A Plug and Play Geometric Loss for Deep Learning
Abstract
Deep neural networks trained using a softmax layer at
the top and the cross-entropy loss are ubiquitous tools for
image classification. Yet, this does not naturally enforce
intra-class similarity nor inter-class margin of the learned
deep representations. To simultaneously achieve these two
goals, different solutions have been proposed in the literature, such as the pairwise or triplet losses. However, these
carry the extra task of selecting pairs or triplets, and the
extra computational burden of computing and learning for
many combinations of them. In this paper, we propose a
plug-and-play loss term for deep networks that explicitly
reduces intra-class variance and enforces inter-class margin simultaneously, in a simple and elegant geometric manner. For each class, the deep features are collapsed into a
learned linear subspace, or union of them, and inter-class
subspaces are pushed to be as orthogonal as possible. Our
proposed Orthogonal Low-rank Embedding (OLE) does not ´
require carefully crafting pairs or triplets of samples for
training, and works standalone as a classification loss, being the first reported deep metric learning framework of its
kind. Because of the improved margin between features of
different classes, the resulting deep networks generalize better, are more discriminative, and more robust. We demonstrate improved classification performance in general object recognition, plugging the proposed loss term into existing off-the-shelf architectures. In particular, we show the
advantage of the proposed loss in the small data/model scenario, and we significantly advance the state-of-the-art on
the Stanford STL-10 benchmark