SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks
for Image Captioning
Abstract
Visual attention has been successfully applied in structural prediction tasks such as visual captioning and question answering. Existing visual attention models are generally spatial, i.e., the attention is modeled as spatial probabilities that re-weight the last conv-layer feature map of
a CNN encoding an input image. However, we argue that
such spatial attention does not necessarily conform to the
attention mechanism — a dynamic feature extractor that
combines contextual fixations over time, as CNN features
are naturally spatial, channel-wise and multi-layer. In this
paper, we introduce a novel convolutional neural network
dubbed SCA-CNN that incorporates Spatial and Channelwise Attentions in a CNN. In the task of image captioning,
SCA-CNN dynamically modulates the sentence generation
context in multi-layer feature maps, encoding where (i.e.,
attentive spatial locations at multiple layers) and what (i.e.,
attentive channels) the visual attention is. We evaluate the
proposed SCA-CNN architecture on three benchmark image
captioning datasets: Flickr8K, Flickr30K, and MSCOCO.
It is consistently observed that SCA-CNN significantly outperforms state-of-the-art visual attention-based image captioning methods