Hint: For there are grayscale images in the dataset, I write a script to remove these images
Development Environment Python 3.5.1 CUDA 8.0
Result
I just train this model for 3 epochs while 11 epochs in the paper, so I think it will work better if train it more.
Good results
Bad results For this network is trained by landscape image database, it's work well
for scenery pictures. So if you use this network to color images of
other types, maybe you can't get a satisfying output.