Rethinking Spatiotemporal Feature Learning:
Speed-Accuracy Trade-offs in Video Classification
Abstract. Despite the steady progress in video analysis led by the adoption of
convolutional neural networks (CNNs), the relative improvement has been less
drastic as that in 2D static image classification. Three main challenges exist including spatial (image) feature representation, temporal information representation, and model/computation complexity. It was recently shown by Carreira and
Zisserman that 3D CNNs, inflated from 2D networks and pretrained on ImageNet, could be a promising way for spatial and temporal representation learning. However, as for model/computation complexity, 3D CNNs are much more
expensive than 2D CNNs and prone to overfit. We seek a balance between speed
and accuracy by building an effective and efficient video classification system
through systematic exploration of critical network design choices. In particular,
we show that it is possible to replace many of the 3D convolutions by low-cost
2D convolutions. Rather surprisingly, best result (in both speed and accuracy) is
achieved when replacing the 3D convolutions at the bottom of the network, suggesting that temporal representation learning on high-level “semantic” features
is more useful. Our conclusion generalizes to datasets with very different properties. When combined with several other cost-effective designs including separable
spatial/temporal convolution and feature gating, our system results in an effective
video classification system that that produces very competitive results on several
action classification benchmarks (Kinetics, Something-something, UCF101 and
HMDB), as well as two action detection (localization) benchmarks (JHMDB and
UCF101-24)