Abstract

Recent applications of Convolutional Neural Networks(ConvNets) for human action recognition in videos haveproposed different solutions for incorporating the appearance and motion information. We study a number of waysof fusing ConvNet towers both spatially and temporally in order to best take advantage of this spatio-temporal infor-mation. We make the following findings: (i) that ratherthan fusing at the softmax layer, a spatial and temporalnetwork can be fused at a convolution layer without loss of performance, but with a substantial saving in parameters; (ii) that it is better to fuse such networks spatially atthe last convolutional layer than earlier, and that additionally fusing at the class prediction layer can boost accuracy;finally (iii) that pooling of abstract convolutional features over spatiotemporal neighbourhoods further boosts performance. Based on these studies we propose a new ConvNet architecture for spatiotemporal fusion of video snippets, and evaluate its performance on standard benchmarks where this architecture achieves state-of-the-art results.