End-to-end Concept Word Detection
for Video Captioning, Retrieval, and Question Answering
Abstract
We propose a high-level concept word detector that can
be integrated with any video-to-language models. It takes
a video as input and generates a list of concept words as
useful semantic priors for language generation models. The
proposed word detector has two important properties. First,
it does not require any external knowledge sources for training. Second, the proposed word detector is trainable in
an end-to-end manner jointly with any video-to-language
models. To effectively exploit the detected words, we also
develop a semantic attention mechanism that selectively focuses on the detected concept words and fuse them with the
word encoding and decoding in the language model. In order to demonstrate that the proposed approach indeed improves the performance of multiple video-to-language tasks,
we participate in all the four tasks of LSMDC 2016 [18].
Our approach has won three of them, including fill-in-theblank, multiple-choice test, and movie retrieval.