Abstract

Convolutional neural networks (CNNs) have showngreat performance as general feature representations forobject recognition applications. However, for multi-labelimages that contain multiple objects from different cate-gories, scales and locations, global CNN features are notoptimal. In this paper, we incorporate local informationto enhance the feature discriminative power. In particu-lar, we first extract object proposals from each image. Witheach image treated as a bag and object proposals extractedfrom it treated as instances, we transform the multi-labelrecognition problem into a multi-class multi-instance learning problem. Then, in addition to extracting the typical CNN feature representation from each proposal, we pro-pose to make use of ground-truth bounding box annotations(strong labels) to add another level of local information by using nearest-neighbor relationships of local regions to form a multi-view pipeline. The proposed multi-view multiinstance framework utilizes both weak and strong labels effectively, and more importantly it has the generalization ability to even boost the performance of unseen categories by partial strong labels from other categories. Our framework is extensively compared with state-of-the-art handcrafted feature based methods and CNN based methods on two multi-label benchmark datasets. The experimental results validate the discriminative power and the generalization ability of the proposed framework. With strong labels, our framework is able to achieve state-of-the-art results in both datasets.