Abstract. Detecting actions in videos is a challenging task as video is
an information intensive media with complex variations. Existing approaches predominantly generate action proposals for each individual
frame or fixed-length clip independently, while overlooking temporal context across them. Such temporal contextual relations are vital for action
detection as an action is by nature a sequence of movements. This motivates us to leverage the localized action proposals in previous frames
when determining action regions in the current one. Specifically, we
present a novel deep architecture called Recurrent Tubelet Proposal and
Recognition (RTPR) networks to incorporate temporal context for action detection. The proposed RTPR consists of two correlated networks,
i.e., Recurrent Tubelet Proposal (RTP) networks and Recurrent Tubelet
Recognition (RTR) networks. The RTP initializes action proposals of
the start frame through a Region Proposal Network and then estimates
the movements of proposals in next frame in a recurrent manner. The
action proposals of different frames are linked to form the tubelet proposals. The RTR capitalizes on a multi-channel architecture, where in
each channel, a tubelet proposal is fed into a CNN plus LSTM to recurrently recognize action in the tubelet. We conduct extensive experiments
on four benchmark datasets and demonstrate superior results over stateof-the-art methods. More remarkably, we obtain mAP of 98.6%, 81.3%,
77.9% and 22.3% with gains of 2.9%, 4.3%, 0.7% and 3.9% over the best
competitors on UCF-Sports, J-HMDB, UCF-101 and AVA, respectively