Abstract

Mobile product image search aims at identifying a product, or retrieving similar products from a database based on a photo captured from a mobile phone camera. Application of traditional image retrieval methods (e.g. bag-of-words) to mobile visual search has been shown to be effective in identifying duplicate/near-duplicate photos, near-planar and textured ob jects such as landmarks, books/cd covers. However, re- trieving more general product categories is still a challenging research problem due to variations in viewpoint, illumination, scale, the existence of blur and background clutter in the query image, etc. In this paper, we propose a new approach that can simultaneously extract the product in- stance from the query, identify the instance, and retrieve visually similar product images. Based on the observation that good query segmenta- tion helps improve retrieval accuracy and good search results provide good priors for segmentation, we formulate our approach in an iterative scheme to improve both query segmentation and retrieval accuracy. To this end, a weighted ob ject mask voting algorithm is proposed based on a spatially-constrained model, which allows robust localization and seg- mentation of the query ob ject, and achieves significantly better retrieval accuracy than previous methods. We show the effectiveness of our ap- proach by applying it to a large, real-world product image dataset and a new ob ject category dataset.