ANN

This repository is for Asymmetric Non-local Neural Networks for Semantic Segmentation (to appear in ICCV 2019),

by Zhen Zhu, Mengde Xu, Song Bai, Tengteng Huang and Xiang Bai.

The source code is in preparing. We will release as soon as possible.

citation

If you find our paper useful in your research, please consider citing:

@inproceedings{annn,
  author    = {Zhen Zhu and
               Mengde Xu and
               Song Bai and
               Tengteng Huang and
               Xiang Bai},
  title     = {Asymmetric Non-local Neural Networks for Semantic Segmentation},
  booktitle={International Conference on Computer Vision},
  year      = {2019},
  url       = {http://arxiv.org/abs/1908.07678},
}

Table of contents

• Introduction

• Usage

• Results

• Acknowledgement

Introduction

Fig.1 Model Architecture

In this work, we present Asymmetric Non-local Neural Network to semantic segmentation for acquiring long range dependency efficiently and effectively. The whole network is shown in Fig. 1. It can fuse features between different level under a sufficient consideration of inter long range dependencies  with AFNB and refine features in the same level involving the inner long range  dependencies with APNB.

Usage

• Preparation

  python==3.6.8
  gcc==5.4.0
  cuda==9.2

  pip install -r requirements.txt   
  cd ext   
  bash make.sh

• Download dataset and pre-process it with scripts in datasets/seg/preprocess.
  Take Cityscapes for example.

  cd datasets/seg/preprocess/cityscapes

  bash cityscapes_seg_generator.sh

• Add dataset path where you save the preprocessed dataset to data.conf or directly modify the data path in scripts.

• Download model pretrained on imagenet and save it in ./pretrained_models.

• Install required package

• Prepare running env:

1. Download the leftImg8bit.zip and gtFine.zip from cityscapes-dataset.com and extract those.

2. Modify the variable ORI_ROOT_DIR in cityscapes_seg_generator.sh to where you save the extracted dataset.

3. run the preprocessing program.

• Train

  Take Cityscapes for example.
  For training only on train set:

  # you can change 'tag' to anything you want
  bash scripts/seg/cityscapes/run_fs_annn_cityscapes_seg.sh train tag

  For training on both train set and validation set:

  bash scripts/seg/cityscapes/run_fs_annn_cityscapes_seg_test.sh train tag

• Validation
  Download trained model to checkpoints/seg/cityscapes/fs_annn_cityscapes_segohem_latest.pth and test on validation set with single scale.

  bash scripts/seg/cityscapes/run_fs_annn_cityscapes_seg.sh val ohem

  The final output should be:

   classes          IoU      nIoU 
  --------------------------------
  road          : 0.984      nan
  sidewalk      : 0.863      nan
  building      : 0.930      nan
  wall          : 0.521      nan
  fence         : 0.629      nan
  pole          : 0.667      nan
  traffic light : 0.737      nan
  traffic sign  : 0.816      nan
  vegetation    : 0.927      nan
  terrain       : 0.648      nan
  sky           : 0.948      nan
  person        : 0.838    0.000
  rider         : 0.663    0.000
  car           : 0.958    0.000
  truck         : 0.830    0.000
  bus           : 0.894    0.000
  train         : 0.858    0.000
  motorcycle    : 0.669    0.000
  bicycle       : 0.796    0.000
  --------------------------------
  Score Average : 0.799    0.000
  --------------------------------
  
  
  categories       IoU      nIoU
  --------------------------------
  flat          : 0.987      nan
  construction  : 0.936      nan
  object        : 0.735      nan
  nature        : 0.929      nan
  sky           : 0.948      nan
  human         : 0.849    0.000
  vehicle       : 0.945    0.000
  --------------------------------
  Score Average : 0.904    0.000
  --------------------------------

• Test on test set

  Download trained model to checkpoints/seg/cityscapes/fs_annn_cityscapes_segohem_latest.pth and test on test set with multiple scales.

  bash scripts/seg/cityscapes/run_fs_annn_cityscapes_seg_test.sh test ohem
  cd results/seg/cityscapes/fs_annn_cityscapes_segohem/test/label
  zip ../res.zip *
  # submit the result on cityscapes official web page.

• Efficiency Statistics

  For details, please refer to ./efficiency_statics/readme.md.

Results

Tab.1 Comparisons on the test set of Cityscapes with the state-of-the-art methods.(multi scale testing)

Tab.2 Comparisons on the validation set of PASCAL Context with the state-of-the-art methods.(multi scale testing)

Tab.3 Comparisons on the validation set of ADE20k with the state-of-the-art methods.(multi scale testing)

Acknowledgement

We genuinely thank Ansheng You for his kind help and suggestions throughout our work. We also recommend others to implement computer vision algorithm with his framework torchcv.