Abstract. We provide a comprehensive evaluation of salient object detection (SOD) models. Our analysis identifies a serious design bias of
existing SOD datasets which assumes that each image contains at least
one clearly outstanding salient object in low clutter. The design bias
has led to a saturated high performance for state-of-the-art SOD models
when evaluated on existing datasets. The models, however, still perform
far from being satisfactory when applied to real-world daily scenes. Based
on our analyses, we first identify 7 crucial aspects that a comprehensive
and balanced dataset should fulfill. Then, we propose a new high quality
dataset and update the previous saliency benchmark. Specifically, our
SOC (Salient Objects in Clutter) dataset, includes images with salient and non-salient objects from daily object categories. Beyond object
category annotations, each salient image is accompanied by attributes
that reflect common challenges in real-world scenes. Finally, we report
attribute-based performance assessment on our dataset