neuron-selectivity-transfer

This code is a re-implementation of the imagenet classification experiments in the paper Like What You Like: Knowledge Distill via Neuron Selectivity Transfer .

Citation

If you use our code in your research or wish to refer to the baseline results, please use the following BibTeX entry.

@article{NST2017
  author =   {Zehao Huang and Naiyan Wang},
  title =    {Like What You Like: Knowledge Distill via Neuron Selectivity Transfer},
  journal =  {arXiv preprint arXiv:1707.01219},
  year =     {2017}
}

Implementation

This code is implemented by a modified MXNet which supports ResNeXt-like augmentation. (This version of MXNet does not support cudnn7)

ImageNet data preparation

Download the ImageNet dataset and create pass through rec (following tornadomeet's repository but using unchange mode)

Run

• bash init.sh

• modify config/cfgs.py

• python train.py

Results on ImageNet-1k

Single Model, Single Crop Validation Error:

| Method | Model | Top-1 | Top-5| Download | | :------------- |:-------------| :-----:| :-----:| :-----:| | Student | Inception-BN | 25.74 | 8.07 | Dropbox | | NST (Poly kernel) | Inception-BN | 24.81 | 7.55 | Dropbox | | NST (Poly kernel) + KD | Inception-BN | 24.34 | 7.11 | Dropbox | | NST (Poly kernel) + KD | A modified ResNet-50 | 21.05 | 5.56 | Dropbox

Note: The symbol of our modified ResNet-50 (following SENet Appendix B) is available in Dropbox. The mean RGB for our modified ResNet-50 is [123.68, 116.28, 103.53], std is [58.395, 57.12, 57.375]. For Inception-BN, we don't need to do data pre-processing since we add a BN layer in the beginning of the network.