Abstract

Most state-of-the-art action feature extractors involvedifferential operators, which act as highpass filters and tendto attenuate low frequency action information. This atten-uation introduces bias to the resulting features and gener-ates ill-conditioned feature matrices. The Gaussian Pyramid has been used as a feature enhancing technique that en-codes scale-invariant characteristics into the feature spacein an attempt to deal with this attenuation. However, at thecore of the Gaussian Pyramid is a convolutional smooth-ing operation, which makes it incapable of generating newfeatures at coarse scales. In order to address this prob-lem, we propose a novel feature enhancing technique calledMulti-skIp Feature Stacking (MIFS), which stacks featuresextracted using a family of differential filters parameter-ized with multiple time skips and encodes shift-invarianceinto the frequency space. MIFS compensates for information lost from using differential operators by recapturinginformation at coarse scales. This recaptured information allows us to match actions at different speeds and ranges of motion. We prove that MIFS enhances the learnability of differential-based features exponentially. The resulting feature matrices from MIFS have much smaller conditional numbers and variances than those from conventional methods. Experimental results show significantly improved per-formance on challenging action recognition and event de-tection tasks. Specifically, our method exceeds the stateof-the-arts on Hollywood2, UCF101 and UCF50 datasets and is comparable to state-of-the-arts on HMDB51 and Olympics Sports datasets. MIFS can also be used as a speedup strategy for feature extraction with minimal or no accuracy cost.