Abstract

We consider ob ject recognition as the process of attaching meaningful labels to specific regions of an image, and propose a model that learns spatial relationships between ob jects. Given a set of images and their associated text (e.g. keywords, captions, descriptions), the ob- jective is to segment an image, in either a crude or sophisticated fashion, then to find the proper associations between words and regions. Previous models are limited by the scope of the representation. In particular, they fail to exploit spatial context in the images and words. We develop a more expressive model that takes this into account. We formulate a spa- tially consistent probabilistic mapping between continuous image feature vectors and the supplied word tokens. By learning both word-to-region associations and ob ject relations, the proposed model augments scene segmentations due to smoothing implicit in spatial consistency. Context introduces cycles to the undirected graph, so we cannot rely on a straight- forward implementation of the EM algorithm for estimating the model parameters and densities of the unknown alignment variables. Instead, we develop an approximate EM algorithm that uses loopy belief propa- gation in the inference step and iterative scaling on the pseudo-likelihood approximation in the parameter update step. The experiments indicate that our approximate inference and learning algorithm converges to good local solutions. Experiments on a diverse array of images show that spa- tial context considerably improves the accuracy of ob ject recognition. Most significantly, spatial context combined with a nonlinear discrete ob- ject representation allows our models to cope well with over-segmented scenes.