Attention Clusters: Purely Attention Based
Local Feature Integration for Video Classification
Abstract
Recently, substantial research effort has focused on how
to apply CNNs or RNNs to better capture temporal patterns
in videos, so as to improve the accuracy of video classifi-
cation. In this paper, however, we show that temporal information, especially longer-term patterns, may not be necessary to achieve competitive results on common trimmed
video classification datasets. We investigate the potential of
a purely attention based local feature integration. Accounting for the characteristics of such features in video classi-
fication, we propose a local feature integration framework
based on attention clusters, and introduce a shifting operation to capture more diverse signals. We carefully analyze
and compare the effect of different attention mechanisms,
cluster sizes, and the use of the shifting operation, and also
investigate the combination of attention clusters for multimodal integration. We demonstrate the effectiveness of our
framework on three real-world video classification datasets.
Our model achieves competitive results across all of these.
In particular, on the large-scale Kinetics dataset, our framework obtains an excellent single model accuracy of 79.4%
in terms of the top-1 and 94.0% in terms of the top-5 accuracy on the validation set.