Abstract

We focus on learning open-vocabulary visual classifiers, which scale up to a large portion of natural language vocabulary (e.g., over tens of thousands of classes). In partic-ular, the training data are large-scale weakly labeled Webimages since it is difficult to acquire sufficient well-labeled data at this category scale. In this paper, we propose a novel online learning paradigm towards this challenging task. Different from traditional N-way independent clas-sifiers that generally fail to handle the extremely sparse and inter-related labels, our classifiers learn from continuous label embeddings discovered by collaboratively decomposing the sparse image-label matrix. Leveraging on the structure of the proposed collaborative learning formulation, we develop an efficient online algorithm that canjointly learn the label embeddings and visual classifiers. The algorithm can learn over 30,000 classes of 1,000 train-ing images within 1 second on a standard GPU. Extensively experimental results on four benchmarks demonstrate the effectiveness of our method.