Abstract
We address the problem of temporal action localization in videos. We pose action localization as a structured
prediction over arbitrary-length temporal windows, where
each window is scored as the sum of frame-wise classifi-
cation scores. Additionally, our model classifies the start,
middle, and end of each action as separate components, allowing our system to explicitly model each action’s temporal evolution and take advantage of informative temporal
dependencies present in this structure. In this framework,
we localize actions by searching for the structured maximal
sum, a problem for which we develop a novel, provablyefficient algorithmic solution. The frame-wise classification
scores are computed using features from a deep Convolutional Neural Network (CNN), which are trained end-toend to directly optimize for a novel structured objective. We
evaluate our system on the THUMOS ’14 action detection
benchmark and achieve competitive performance.