Abstract
The attention mechanisms in deep neural networks are
inspired by human’s attention that sequentially focuses on
the most relevant parts of the information over time to generate prediction output. The attention parameters in those
models are implicitly trained in an end-to-end manner, yet
there have been few trials to explicitly incorporate human
gaze tracking to supervise the attention models. In this paper, we investigate whether attention models can benefit
from explicit human gaze labels, especially for the task of
video captioning. We collect a new dataset called VAS, consisting of movie clips, and corresponding multiple descriptive sentences along with human gaze tracking data. We
propose a video captioning model named Gaze Encoding
Attention Network (GEAN) that can leverage gaze tracking information to provide the spatial and temporal attention for sentence generation. Through evaluation of language similarity metrics and human assessment via Amazon mechanical Turk, we demonstrate that spatial attentions guided by human gaze data indeed improve the performance of multiple captioning methods. Moreover, we show
that the proposed approach achieves the state-of-the-art
performance for both gaze prediction and video captioning
not only in our VAS dataset but also in standard datasets
(e.g. LSMDC [24] and Hollywood2 [18]).