AdaScan: Adaptive Scan Pooling in Deep Convolutional Neural Networks
for Human Action Recognition in Videos
Abstract
We propose a novel method for temporally pooling
frames in a video for the task of human action recognition. The method is motivated by the observation that there
are only a small number of frames which, together, contain sufficient information to discriminate an action class
present in a video, from the rest. The proposed method
learns to pool such discriminative and informative frames,
while discarding a majority of the non-informative frames
in a single temporal scan of the video. Our algorithm does
so by continuously predicting the discriminative importance
of each video frame and subsequently pooling them in a
deep learning framework. We show the effectiveness of our
proposed pooling method on standard benchmarks where
it consistently improves on baseline pooling methods, with
both RGB and optical flow based Convolutional networks.
Further, in combination with complementary video representations, we show results that are competitive with respect
to the state-of-the-art results on two challenging and publicly available benchmark datasets