Abstract. In this paper, we aim to reduce the computational cost of
spatio-temporal deep neural networks, making them run as fast as their
2D counterparts while preserving state-of-the-art accuracy on video recognition benchmarks. To this end, we present the novel Multi-Fiber architecture that slices a complex neural network into an ensemble of
lightweight networks or fibers that run through the network. To facilitate information flow between fibers we further incorporate multiplexer
modules and end up with an architecture that reduces the computational
cost of 3D networks by an order of magnitude, while increasing recognition performance at the same time. Extensive experimental results show
that our multi-fiber architecture significantly boosts the efficiency of existing convolution networks for both image and video recognition tasks,
achieving state-of-the-art performance on UCF-101, HMDB-51 and Kinetics datasets. Our proposed model requires over 9× and 13× less computations than the I3D [1] and R(2+1)D [2] models, respectively, yet
providing higher accuracy