Abstract
Semantic classes can be either things (objects with a
well-defined shape, e.g. car, person) or stuff (amorphous
background regions, e.g. grass, sky). While lots of classifi-
cation and detection works focus on thing classes, less attention has been given to stuff classes. Nonetheless, stuff
classes are important as they allow to explain important
aspects of an image, including (1) scene type; (2) which
thing classes are likely to be present and their location
(through contextual reasoning); (3) physical attributes, material types and geometric properties of the scene. To understand stuff and things in context we introduce COCOStuff 1
, which augments all 164K images of the COCO 2017
dataset with pixel-wise annotations for 91 stuff classes. We
introduce an efficient stuff annotation protocol based on superpixels, which leverages the original thing annotations.
We quantify the speed versus quality trade-off of our protocol and explore the relation between annotation time and
boundary complexity. Furthermore, we use COCO-Stuff to
analyze: (a) the importance of stuff and thing classes in
terms of their surface cover and how frequently they are
mentioned in image captions; (b) the spatial relations between stuff and things, highlighting the rich contextual relations that make our dataset unique; (c) the performance of
a modern semantic segmentation method on stuff and thing
classes, and whether stuff is easier to segment than things