Abstract

While recent deep neural network models have achieved promising results on the image captioning task, they relylargely on the availability of corpora with paired image and sentence captions to describe objects in context. In this work, we propose the Deep Compositional Captioner (DCC) to address the task of generating descriptions of novel objects which are not present in paired imagesentence datasets. Our method achieves this by leveraginglarge object recognition datasets and external text corporaand by transferring knowledge between semantically similar concepts. Current deep caption models can only de-scribe objects contained in paired image-sentence corpora,despite the fact that they are pre-trained with large objectrecognition datasets, namely ImageNet. In contrast, our model can compose sentences that describe novel objects and their interactions with other objects. We demonstrate our model’s ability to describe novel concepts by empirically evaluating its performance on MSCOCO and showqualitative results on ImageNet images of objects for which no paired image-sentence data exist. Further, we extend our approach to generate descriptions of objects in video clips. Our results show that DCC has distinct advantages over existing image and video captioning approaches for generating descriptions of new objects in context.