Abstract
Representations that can compactly and effectively capture the temporal evolution of semantic content are important
to computer vision and machine learning algorithms that operate on multi-variate time-series data. We investigate such
representations motivated by the task of human action recognition. Here each data instance is encoded by a multivariate
feature (such as via a deep CNN) where action dynamics are
characterized by their variations in time. As these features
are often non-linear, we propose a novel pooling method,
kernelized rank pooling, that represents a given sequence
compactly as the pre-image of the parameters of a hyperplane in a reproducing kernel Hilbert space, projections of
data onto which captures their temporal order. We develop
this idea further and show that such a pooling scheme can be
cast as an order-constrained kernelized PCA objective. We
then propose to use the parameters of a kernelized low-rank
feature subspace as the representation of the sequences. We
cast our formulation as an optimization problem on generalized Grassmann manifolds and then solve it efficiently using
Riemannian optimization techniques. We present experiments on several action recognition datasets using diverse
feature modalities and demonstrate state-of-the-art results.