Spatiotemporal Residual Networks for Video Action Recognition

This repository contains the code for our NIPS'16 and CVPR'17 papers:

Christoph Feichtenhofer, Axel Pinz, Richard P. Wildes
"Spatiotemporal Residual Networks for Video Action Recognition"
in Proc. NIPS 2016

Christoph Feichtenhofer, Axel Pinz, Richard P. Wildes
"Spatiotemporal Multiplier Networks for Video Action Recognition"
in Proc. CVPR 2017

If you use our code/models/data for your research, please cite our papers:

    @inproceedings{feichtenhofer2016spatiotemporal,
      title={Spatiotemporal residual networks for video action recognition},
      author={Feichtenhofer, Christoph and Pinz, Axel and Wildes, Richard},
      booktitle={Advances in Neural Information Processing Systems (NIPS)},
      pages={3468--3476},
      year={2016}
    }

    @inproceedings{feichtenhofer2017multiplier,
      title={Spatiotemporal multiplier networks for video action recognition},
      author={Feichtenhofer, Christoph and Pinz, Axel and Wildes, Richard P}
      booktitle={Conference on Computer Vision and Pattern Recognition (CVPR)},
      year={2017}
    }

Requirements

The code was tested on Ubuntu 14.04, 16.04 and Windows 10 using MATLAB R2016b and NVIDIA Titan X GPUs.

If you have questions regarding the implementation please contact:

Christoph Feichtenhofer <feichtenhofer AT tugraz.at>

================================================================================

Setup

1. Download the code git clone --recursive https://github.com/feichtenhofer/st-resnet

2. Compile the code by running compile.m.

• This will also compile our own branch of the MatConvNet toolbox. In case of any issues, please follow the installation instructions on the MatConvNet homepage.

3. Edit the file cnn_setup_environment.m to adjust the models and data paths.

4. (Optional) Download pretrained model files and the datasets, linked below and unpack them into your models/data directory. Otherwise the scripts will attempt to download the models at runtime.

• Optionally you can also pretrain your own twostream base models by running

1. base_streams/cnn_ucf101_spatial(); to train the appearance network streams.

2. base_streams/cnn_ucf101_temporal(); to train the optical flow network streams.

5. Training STResNet_stage1();, STResNet_stage2(); to train the architecture in our NIPS 2016 paper. STMulNet(); to train the architecture in our CVPR 2017 paper.

• In case you did not download or trained the base models, the script will attempt to download these accordingly.

• In case you would like to train on the CPU, clear the variable opts.train.gpus

• In case you encounter memory issues on your GPU, consider decreasing the cudnnWorkspaceLimit (512MB is default)

Models: ST-ResNet

• Download final models here: (in case you do not, STResNet_stage1();, STResNet_stage2();, and STResNet_test(); will attempt to download the respective models at runtime.)

• STResNet_base

• STResNet_stage1

• STResNet_stage2

Models: ST-MulNet

• Download final models here: (in case you do not, STMulNet(); and STMulNet_test(); will attempt to download the models at runtime.)

• STMulNet_base

• STMulNet_final

Data

Pre-computed optical flow images and resized rgb frames for the UCF101 and HMDB51 datasets

• UCF101 RGB: part1 part2 part3

• UCF101 Flow: part1 part2 part3

• HMDB51 RGB: part1

• HMDB51 Flow: part1

Use it on your own dataset

• Our Optical flow extraction tool provides OpenCV wrappers for optical flow extraction on a GPU.