Abstract. Despite the recent progress in video understanding and the
continuous rate of improvement in temporal action localization throughout the years, it is still unclear how far (or close?) we are to solving the
problem. To this end, we introduce a new diagnostic tool to analyze the
performance of temporal action detectors in videos and compare different
methods beyond a single scalar metric. We exemplify the use of our tool
by analyzing the performance of the top rewarded entries in the latest ActivityNet action localization challenge. Our analysis shows that the most
impactful areas to work on are: strategies to better handle temporal context around the instances, improving the robustness w.r.t. the instance
absolute and relative size, and strategies to reduce the localization errors.
Moreover, our experimental analysis finds the lack of agreement among
annotator is not a major roadblock to attain progress in the field. Our
diagnostic tool is publicly available to keep fueling the minds of other
researchers with additional insights about their algorithms