Abstract We present an optical ?ow algorithm for large displacement motions. Most existing optical ?ow methods use the standard coarse-to-?ne framework to deal with large displacement motions which has intrinsic limitations. Instead, we formulate the motion estimation problem as a motion segmentation problem. We use approximate nearest neighbor ?elds to compute an initial motion ?eld and use a robust algorithm to compute a set of similarity transformations as the motion candidates for segmentation. To account for deviations from similarity transformations, we add local deformations in the segmentation process. We also observe that small objects can be better recovered using translation s as the motion candidates. We fuse the motion results obtained under similarity transformations and under translations together before a ?nal re?nement. Experimental validation shows that our method can successfully handle large displacement motions. Although we particularly focus on large displacement motions in this work, we make no sacri?ce in terms of overall performance. In particular, our method ranks at the top of the Middlebury benchmark.