Video Captioning

Dependencies

• Python 2.7

• Pytorch 0.2

• Microsoft COCO Caption Evaluation

• CIDEr

• torch, torchvision, numpy, scikit-image, nltk, h5py, pandas, future

• tensorboard_logger--for use tensorboard to view training loss

(Check out the coco-caption and cider projects into your working directory)

Data

• VGG16 pretrained on ImageNet [PyTorch version]: https://download.pytorch.org/models/vgg16-397923af.pth

• Resnet-101 pretrained on ImageNet [PyTorch version]: https://github.com/ruotianluo/pytorch-resnet

• MSVD: https://www.microsoft.com/en-us/download/details.aspx?id=52422

• MSR-VTT: http://ms-multimedia-challenge.com/2017/dataset

• Flickr30k: flickr30k.tar.gz, flickr30k-images.tar

Obtain the dataset you need:

• MSR-VTT:train_val_videos.zip,train_val_annotation.zip,test_videos.zip,test_videodatainfo.json

Getting started

Generate metadata

1. run func_standalize_format

2. run func_preprocess_datainfo

3. run func_build_vocab

4. run func_create_sequencelabel

5. run func_convert_datainfo2cocofmt

6. run func_compute_ciderdf # Pre-compute document frequency for CIDEr computation

7. run func_compute_evalscores # Pre-compute evaluation scores (BLEU_4, CIDEr, METEOR, ROUGE_L) for each caption

8. run func_extract_video_features # extract video features

Training

Please refer to the opts.py file for the set of available train/test options

# Train XE model./train.sh 0 [GPUIDs]

# Train CST_GT_None/WXE model./train.sh 1 [GPUIDs]

# Train CST_MS_Greedy model (using greedy baseline)./train.sh 2 [GPUIDs]

# Train CST_MS_SCB model (using SCB baseline, where SCB is computed from GT captions)./train.sh 3 [GPUIDs]

#Train CST_MS_SCB(*) model (using SCB baseline, where SCB is computed from model sampled captions)./train.sh 4 [GPUIDs]

Testing

./test.sh 0 [GPUIDs]

Acknowledgements

• Torch implementation of NeuralTalk2

• PyTorch implementation of Self-critical Sequence Training for Image Captioning (SCST)

• PyTorch Team

• "Consensus-based Sequence Training for Video Captioning" (Phan, Henter, Miyao, Satoh. 2017).