Abstract

We describe a novel framework for the joint processing of color and shape information in natural images. A hierarchical non-linear spatio-chromatic operator yields spatial and chromatic opponent chan- nels, which mimics processing in the primate visual cortex. We extend two popular ob ject recognition systems (i.e., the Hmax hierarchical model of visual processing and a sift-based bag-of-words approach) to incorpo- rate color information along with shape information. We further use the framework in combination with the gist algorithm for scene categoriza- tion as well as the Berkeley segmentation algorithm. In all cases, the pro- posed approach is shown to outperform standard grayscale/shape-based descriptors as well as alternative color processing schemes on several datasets.