Abstract

In this work we improve training of temporal deep mod-els to better learn activity progression for activity detec-tion and early detection tasks. Conventionally, when train-ing a Recurrent Neural Network, specifically a Long ShortTerm Memory (LSTM) model, the training loss only consid-ers classification error. However, we argue that the detec-tion score of the correct activity category, or the detectionscore margin between the correct and incorrect categories,should be monotonically non-decreasing as the model ob-serves more of the activity. We design novel ranking losses that directly penalize the model on violation of such monotonicities, which are used together with classification loss intraining of LSTM models. Evaluation on ActivityNet shows significant benefits of the proposed ranking losses in both activity detection and early detection tasks.