Abstract

We introduce the self-similar sketch, a new method for the extraction of intermediate image features that combines three princi- ples: detection of self-similarity structures, nonaccidental alignment, and instance-specific modelling. The method searches for self-similar image structures that form nonaccidental patterns, for example collinear ar- rangements. We demonstrate a simple implementation of this idea where self-similar structures are found by looking for SIFT descriptors that map to the same visual words in image-specific vocabularies. This results in a visual word map which is searched for elongated connected compo- nents. Finally, segments are fitted to these connected components, ex- tracting linear image structures beyond the ones that can be captured by conventional edge detectors, as the latter implicitly assume a specific appearance for the edges (steps). The resulting collection of segments constitutes a “sketch” of the image. This is applied to the task of es- timating vanishing points, horizon, and zenith in standard benchmark data, obtaining state-of-the-art results. We also propose a new vanish- ing point estimation algorithm based on recently introduced techniques for the continuous-discrete optimisation of energies arising from model selection priors.