Abstract

We learn to compute optical flflow by combining a classical spatial-pyramid formulation with deep learning. This estimates large motions in a coarse-to-fifine approach by warping one image of a pair at each pyramid level by the current flflow estimate and computing an update to the flflow. Instead of the standard minimization of an objective function at each pyramid level, we train one deep network per level to compute the flflow update. Unlike the recent FlowNet approach, the networks do not need to deal with large motions; these are dealt with by the pyramid. This has several advantages. First, our Spatial Pyramid Network (SPyNet) is much simpler and 96% smaller than FlowNet in terms of model parameters. This makes it more effificient and appropriate for embedded applications. Second, since the flflow at each pyramid level is small (< 1 pixel), a convolutional approach applied to pairs of warped images is appropriate. Third, unlike FlowNet, the learned convolution fifilters appear similar to classical spatio-temporal fifilters, giving insight into the method and how to improve it. Our results are more accurate than FlowNet on most standard benchmarks, suggesting a new direction of combining classical flflow methods with deep learning