Serving PyTorch Models in C++ on Windows10 platforms

How to use

Prepare Data

examples/data/train/

	- 0
	- 1
	  .
	  .
	  .
	- n

examples/data/test/

	- 0
	- 1
	  .
	  .
	  .
	- n

Train Model

cd examples && python train.py

Transform Model

cd examples && python transform_model.py

Test Model

cd makefile/pytorch
mkdir build && cd build && cmake -A x64 ..

or

mkdir build && cd build && cmake -G "Visual Studio 15 2017 Win64" ..

set Command Arguments -> ......examplescheckpoint ......examplesimages
set Environment -> path=%path%;../../../thirdparty/libtorch/lib;../../../thirdparty/opencv/build/x64/vc15/bin;

Test CUDA Softmax

cd makefile/cuda
mkdir build && cd build && cmake -A x64 ..

or

mkdir build && cd build && cmake -G "Visual Studio 15 2017 Win64" ..

Inference onnx model

cd makefile/tensorRT/classification
mkdir build && cd build && cmake -A x64 ..

or

mkdir build && cd build && cmake -G "Visual Studio 15 2017 Win64" ..
set Environment -> path=%path%;../../../../thirdparty/TensorRT/lib;

Inference caffe model for faster-rcnn

cd makefile/tensorRT/detection
mkdir build && cd build && cmake -A x64 ..

or

mkdir build && cd build && cmake -G "Visual Studio 15 2017 Win64" ..
set Environment -> path=%path%;../../../../thirdparty/TensorRT/lib;

download VGG16_faster_rcnn_final.caffemodel

Thirdparty

thirdparty/
	- libtorch  
	- opencv 
	- CUDA
	- TensorRT

download thirdparty from here.

Docker

docker pull zccyman/deepframe
nvidia-docker run -it --name=mydocker zccyman/deepframe /bin/bash
cd workspace && git clone https://github.com/zccyman/pytorch-inference.git

Environment

• Windows10

• VS2017

• CMake3.13

• CUDA10.0

• CUDNN7.3

• Pyton3.5

• ONNX1.1.2

• TensorRT5.0.1

• Pytorch1.0

• Libtorch

• OpenCV4.0.1

Todo List

•  train and transform pytorch model

•  multi-batch inference pytorch model in C++

•  cpu and gpu softmax

•  transform pytorch model to ONNX model, and inference onnx model using tensorRT

•  inference caffe model for faster-rcnn using tensorRT

•  build classification network

•  compress pytorch model

•  object detection pytorch inference using C++ on Window platforms

Notes

• "torch.jit.trace" doesn’t support nn.DataParallel so far.

• TensorRT doesn’t supports opset 7 above so far, but pyTorch ONNX exporter seems to export opset 9.

Acknowledgement

• https://github.com/Wizaron/pytorch-cpp-inference

• https://zhuanlan.zhihu.com/p/52154049

• Caffe

• Pytorch

• TensorRT