Abstract
Despite recent emergence of video caption methods, how
to generate fine-grained video descriptions (i.e., long and
detailed commentary about individual movements of multiple subjects as well as their frequent interactions) is far
from being solved, which however has great applications
such as automatic sports narrative. To this end, this work
makes the following contributions. First, to facilitate this
novel research of fine-grained video caption, we collected a
novel dataset called Fine-grained Sports Narrative dataset
(FSN) that contains 2K sports videos with ground-truth narratives from YouTube.com. Second, we develop a novel performance evaluation metric named Fine-grained Captioning Evaluation (FCE) to cope with this novel task. Considered as an extension of the widely used METEOR, it measures not only the linguistic performance but also whether
the action details and their temporal orders are correctly
described. Third, we propose a new framework for finegrained sports narrative task. This network features three
branches: 1) a spatio-temporal entity localization and role
discovering sub-network; 2) a fine-grained action modeling
sub-network for local skeleton motion description; and 3)
a group relationship modeling sub-network to model interactions between players. We further fuse the features and
decode them into long narratives by a hierarchically recurrent structure. Extensive experiments on the FSN dataset
demonstrates the validity of the proposed framework for
fine-grained video caption