About this fork

This fork contains pure python version of rmpe_dataset_server. It have less code(19kb vs 35kb), and significantly faster (140 images/s vs 30 images/s C++ code on my machine) Could be run as iterator inside train_pose.py (default), or as ./rmpe_server.py

Current status

• [x] image augmentation: rotate, shift, scale, crop, flip (implemented as single affine transform, i.e. much faster)

• [x] mask calculation: rotate, shift, scale, crop, flip

• [x] joint heatmaps

• [x] limbs part affinity fields

• [x] tested using rmpe_server_tester.py, found some differences from C++ version, but looks like it is C++ code is buggy

How to help

• re-generate val_dataset.h5 with new version of generate_hdf5.py (will be backward compatible, just one attribute 'meta' added)

• since augmentation is very fast now, by default it works inside train_pose.py (separate thread)

• if you want to run external augmentation server run ./rmpe_server.py and change use_client_gen = True in train_pose.py

• test result with inspect_dataset.ipynb or rmpe_server_tester.py raw save (saves all images, heatmaps and PAFs to disk). Make sure to initialize the server python rmpe_server.py

• look to the code and give feedback

• try to train

Realtime Multi-Person Pose Estimation

This is a keras version of Realtime Multi-Person Pose Estimation project

Introduction

Code repo for reproducing 2017 CVPR paper using keras.

Results

Contents

1. Converting caffe model

2. Testing

3. Training

Require

1. Keras

2. Caffe - docker required if you would like to convert caffe model to keras model. You don't have to compile/install caffe on your local machine.

Converting Caffe model to Keras model

Authors of original implementation released already trained caffe model which you can use to extract weights data.

• Download caffe model cd model; sh get_caffe_model.sh

• Dump caffe layers to numpy data cd ..; docker run -v [absolute path to your keras_Realtime_Multi-Person_Pose_Estimation folder]:/workspace -it bvlc/caffe:cpu python dump_caffe_layers.py Note that docker accepts only absolute paths so you have to set the full path to the folder containing this project.

• Convert caffe model (from numpy data) to keras model python caffe_to_keras.py

Testing steps

• Convert caffe model to keras model or download already converted keras model https://www.dropbox.com/s/llpxd14is7gyj0z/model.h5

• Run the notebook demo.ipynb.

• python demo_image.py --image sample_images/ski.jpg to run the picture demo. Result will be stored in the file result.png. You can use any image file as an input.

• python demo_camera.py to run the web demo.

Training steps

UPDATE 26/10/2017

Fixed problem with the training procedure. Here are my results after training for 5 epochs = 25000 iterations (1 epoch is ~5000 batches) The loss values are quite similar as in the original training - output.txt

Results of running demo_image --image sample_images/ski.jpg --model training/weights.best.h5 with model trained only 25000 iterations. Not too bad !!! Training on my single 1070 GPU took around 10 hours.

UPDATE 22/10/2017:

Augmented samples are fetched from the server. The network never sees the same image twice which was a problem in previous approach (tool rmpe_dataset_transformer) This allows you to run augmentation locally or on separate node. You can start 2 instances, one serving training set and a second one serving validation set (on different port if locally)

• Install gsutil curl https://sdk.cloud.google.com | bash. This is a really helpful tool for downloading large datasets.

• Download the data set (~25 GB) cd dataset; sh get_dataset.sh,

• Download COCO official toolbox in dataset/coco/ .

• cd coco/PythonAPI; sudo python setup.py install to install pycocotools.

• Go to the "training" folder cd ../../../training.

• Generate masks python generate_masks.py. Note: set the parameter "mode" in generate_masks.py (validation or training)

• Create intermediate dataset python generate_hdf5.py. This tool creates a dataset in hdf5 format. The structure of this dataset is very similar to the original lmdb dataset where a sample is represented as an array: 5 x width x height (3 channels for image, 1 channel for metedata, 1 channel for miss masks) For MPI dataset there are 6 channels with additional all masks. Note: set the parameters datasets and val_size in generate_hdf5.py

• Download and compile the dataset server rmpe_dataset_server. This server generates augmented samples on the fly. Source samples are retrieved from previously generated hdf5 dataset file.

• Start training data server in the first terminal session. ./rmpe_dataset_server ../../keras_Realtime_Multi-Person_Pose_Estimation/dataset/train_dataset.h5 5555

• Start validation data server in a second terminal session. ./rmpe_dataset_server ../../keras_Realtime_Multi-Person_Pose_Estimation/dataset/val_dataset.h5 5556

• Optionally you can verify the datasets inspect_dataset.ipynb

• Set the correct number of samples within python train_pose.py - variables "train_samples = ???" and "val_samples = ???".
  This number is used by keras to determine how many samples are in 1 epoch.

• Train the model in a third terminal python train_pose.py

Related repository

• CVPR'16, Convolutional Pose Machines.

• CVPR'17, Realtime Multi-Person Pose Estimation.

Citation

Please cite the paper in your publications if it helps your research:

@InProceedings{cao2017realtime,
  title = {Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields},
  author = {Zhe Cao and Tomas Simon and Shih-En Wei and Yaser Sheikh},
  booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2017}
  }