Abstract

Image classification is a critical task for both humans and computers. One of the challenges lies in the large scale of the semantic space. In particular, humans can recognize tens of thousands of ob ject classes and scenes. No computer vision algorithm today has been tested at this scale. This paper presents a study of large scale categorization including a series of challenging experiments on classification with more than 10, 000 image classes. We find that a) computational issues be- come crucial in algorithm design; b) conventional wisdom from a couple of hundred image categories on relative performance of different classi- fiers does not necessarily hold when the number of categories increases; c) there is a surprisingly strong relationship between the structure of WordNet (developed for studying language) and the difficulty of visual categorization; d) classification can be improved by exploiting the se- mantic hierarchy. Toward the future goal of developing automatic vision algorithms to recognize tens of thousands or even millions of image cat- egories, we make a series of observations and arguments about dataset scale, category density, and image hierarchy.