PNASNet.pytorch

PyTorch implementation of PNASNet-5. Specifically, PyTorch code from this repository is adapted to completely match both my implemetation and the official implementation of PNASNet-5, both written in TensorFlow. This complete match allows the pretrained TF model to be exactly converted to PyTorch: see convert.py.

If you use the code, please cite:

@inproceedings{liu2018progressive,  author    = {Chenxi Liu and
               Barret Zoph and
               Maxim Neumann and
               Jonathon Shlens and
               Wei Hua and
               Li{-}Jia Li and
               Li Fei{-}Fei and
               Alan L. Yuille and
               Jonathan Huang and
               Kevin Murphy},  title     = {Progressive Neural Architecture Search},  booktitle = {European Conference on Computer Vision},  year      = {2018}}

Requirements

• TensorFlow 1.8.0 (for image preprocessing)

• PyTorch 0.4.0

• torchvision 0.2.1

Data and Model Preparation

• Download the ImageNet validation set and move images to labeled subfolders. To do the latter, you can use this script. Make sure the folder val is under data/.

• Download PNASNet.TF and follow its README to download the PNASNet-5_Large_331 pretrained model.

• Convert TensorFlow model to PyTorch model:

python convert.py

Notes on Model Conversion

• In both TensorFlow implementations, net[0] means prev and net[1] means prev_prev. However, in the PyTorch implementation, states[0] means prev_prev and states[1] means prev. I followed the PyTorch implemetation in this repository. This is why the 0 and 1 in PNASCell specification are reversed.

• The default value of eps in BatchNorm layers is 1e-3 in TensorFlow and 1e-5 in PyTorch. I changed all BatchNorm eps values to 1e-3 (see operations.py) to exactly match the TensorFlow pretrained model.

• The TensorFlow pretrained model uses tf.image.resize_bilinear to resize the image (see utils.py). I cannot find a python function that exactly matches this function's behavior (also see this thread and this post on this topic), so currently in main.py I call TensorFlow to do the image preprocessing, in order to guarantee both models have the identical input.

• When converting the model from TensorFlow to PyTorch (i.e. convert.py), I use input image size of 323 instead of 331. This is because the 'SAME' padding in TensorFlow may differ from padding in PyTorch in some layers (see this link; basically TF may only pad 1 right and bottom, whereas PyTorch always pads 1 for all four margins). However, they behave exactly the same when image size is 323: conv0 does not have padding, so feature size becomes 161, then 81, 41, etc.

• The exact conversion when image size is 323 is also corroborated by the following table:

Image Size | Official TensorFlow Model | Converted PyTorch Model --- | --- | --- (331, 331) | (0.829, 0.962) | (0.828, 0.961) (323, 323) | (0.827, 0.961) | (0.827, 0.961)

Usage

python main.py

The last printed line should read:

Test: [50000/50000] Prec@1 0.828    Prec@5 0.961