Abstract

The well-known word analogy experiments show that therecent word vectors capture fine-grained linguistic regular-ities in words by linear vector offsets, but it is unclear howwell the simple vector offsets can encode visual regularitiesover words. We study a particular image-word relevancerelation in this paper. Our results show that the word vec-tors of relevant tags for a given image rank ahead of theirrelevant tags, along a principal direction in the word vec-tor space. Inspired by this observation, we propose to solveimage tagging by estimating the principal direction for animage. Particularly, we exploit linear mappings and nonlinear deep neural networks to approximate the principal direction from an input image. We arrive at a quite versatile tagging model. It runs fast given a test image, in constant time w.r.t. the training set size. It not only gives superior performance for the conventional tagging task on the NUSWIDE dataset, but also outperforms competitive baselines on annotating images with previously unseen tags.