Abstract

Saliency in Context (SALICON) is an ongoing effort thataims at understanding and predicting visual attention. Thispaper presents a new method to collect large-scale humandata during natural explorations on images. While currentdatasets present a rich set of images and task-specific an-notations such as category labels and object segments, thiswork focuses on recording and logging how humans shifttheir attention during visual exploration. The goal is to of-fer new possibilities to (1) complement task-specific annota-tions to advance the ultimate goal in visual understanding,and (2) understand visual attention and learn saliency mod-els, all with human attentional data at a much larger scale. We designed a mouse-contingent multi-resolutionalparadigm based on neurophysiological and psychophysicalstudies of peripheral vision, to simulate the natural viewingbehavior of humans. The new paradigm allowed using ageneral-purpose mouse instead of an eye tracker to recordviewing behaviors, thus enabling large-scale data collection. The paradigm was validated with controlled laboratory as well as large-scale online data. We report in this paper a proof-of-concept SALICON dataset of human “freeviewing” data on 10,000 images from the Microsoft COCO (MS COCO) dataset with rich contextual information. We evaluated the use of the collected data in the context of saliency prediction, and demonstrated them a good source as ground truth for the evaluation of saliency algorithms.