Abstract
We propose a new self-supervised CNN pre-training
technique based on a novel auxiliary task called odd-oneout learning. In this task, the machine is asked to identify the
unrelated or odd element from a set of otherwise related elements. We apply this technique to self-supervised video representation learning where we sample subsequences from
videos and ask the network to learn to predict the odd video
subsequence. The odd video subsequence is sampled such
that it has wrong temporal order of frames while the even
ones have the correct temporal order. Therefore, to generate
a odd-one-out question no manual annotation is required.
Our learning machine is implemented as multi-stream convolutional neural network, which is learned end-to-end. Using odd-one-out networks, we learn temporal representations for videos that generalizes to other related tasks such
as action recognition.
On action classification, our method obtains 60.3% on
the UCF101 dataset using only UCF101 data for training which is approximately 10% better than current stateof-the-art self-supervised learning methods. Similarly, on
HMDB51 dataset we outperform self-supervised state-ofthe art methods by 12.7% on action classification task.