Abstract

Even a relatively unstructured captioned image set depict- ing a variety of ob jects in cluttered scenes contains strong correlations between caption words and repeated visual structures. We exploit these correlations to discover named ob jects and learn hierarchical models of their appearance. Revising and extending a previous technique for finding small, distinctive configurations of local features, our method assembles these co-occurring parts into graphs with greater spatial extent and flex- ibility. The resulting multipart appearance models remain scale, transla- tion and rotation invariant, but are more reliable detectors and provide better localization. We demonstrate improved annotation precision and recall on datasets to which the non-hierarchical technique was previously applied and show extended spatial coverage of detected ob jects.