Highway Networks

This is sample code for convolutional Highway Networks, implemented in Caffe checked out at this state. It runs only on the NVIDIA GPUs and requires NVIDIA's cuDNN v2. The original Caffe README is reproduced below the line.

Highway Networks utilize the idea of information highways, in turn inspired by LSTM networks [1, 2, 3]. Our experiments on highway networks show that when designed correctly, neural networks with tens, even hundreds of layers can be trained directly with stochastic gradient descent, thereby providing a promising solution to the vanishing gradient problem. More information is available on the Project Website.

Highway Networks were introduced in the following paper:

Srivastava, R. K., Greff, K., & Schmidhuber, J. (2015). Highway Networks. arXiv preprint arXiv:1505.00387.

Followed by a more detailed report:

Srivastava, R. K., Greff, K., & Schmidhuber, J. (2015). Training Very Deep Networks. arXiv preprint arXiv:1507.06228.

Data

You can get preprocessed datasets (CIFAR-10/100 were global contrast normalized and padded with 8 pixels each side) at the links below. Caffe's data generation scripts do not generate validation sets. The provided data below includes splits into training and validation sets.

CIFAR-10 ~ 2.27 GB

CIFAR-100 ~ 2.27 GB

MNIST ~ 20 MB

Examples

See examples and sample log outputs in examples/highways. You probably need to adjust the paths to the datasets in the network definition files in order to train networks.

Citation

Please cite us if you use this code:

@article{srivastava2015highway,
    title={Training Very Deep Networks},
    author={Srivastava, Rupesh Kumar and Greff, Klaus and Schmidhuber, J{"u}rgen},
    journal={arXiv preprint arXiv:1507.06228},
    year={2015}
}

Caffe

Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by the Berkeley Vision and Learning Center (BVLC) and community contributors.

Check out the project site for all the details like

• DIY Deep Learning for Vision with Caffe

• Tutorial Documentation

• BVLC reference models and the community model zoo

• Installation instructions

and step-by-step examples.

Please join the caffe-users group or gitter chat to ask questions and talk about methods and models. Framework development discussions and thorough bug reports are collected on Issues.

Happy brewing!

License and Citation

Caffe is released under the BSD 2-Clause license. The BVLC reference models are released for unrestricted use.

Please cite Caffe in your publications if it helps your research:

@article{jia2014caffe,
  Author = {Jia, Yangqing and Shelhamer, Evan and Donahue, Jeff and Karayev, Sergey and Long, Jonathan and Girshick, Ross and Guadarrama, Sergio and Darrell, Trevor},
  Journal = {arXiv preprint arXiv:1408.5093},
  Title = {Caffe: Convolutional Architecture for Fast Feature Embedding},
  Year = {2014}
}