Abstract

The so-called bag-of-features (BoF) representation for im- ages is by now well-established in the context of large scale image and video retrieval. The BoF framework typically ranks database image ac- cording to a metric on the global histograms of the query and database images, respectively. Ranking based on global histograms has the advan- tage of being scalable with respect to the number of database images, but at the cost of reduced retrieval precision when the ob ject of interest is small. Additionally, computationally intensive post-processing (such as RANSAC) is typically required to locate the ob ject of interest in the retrieved images. To address these shortcomings, we propose a general- ization of the global BoF framework to support scalable local matching. Specifically, we propose an efficient and accurate algorithm to accom- plish local histogram matching and ob ject localization simultaneously. The generalization is to represent each database image as a family of histograms that depend functionally on a bounding rectangle. Integral with the image retrieval process, we identify bounding rectangles whose histograms optimize query relevance, and rank the images accordingly. Through this localization scheme, we impose a weak spatial consistency constraint with low computational overhead. We validate our approach on two public image retrieval benchmarks: the University of Kentucky data set and the Oxford Building data set. Experiments show that our approach significantly improves on BoF-based retrieval, without requir- ing computationally expensive post-processing.