Abstract. We introduce a novel Recurrent Neural Network-based algorithm for
future video feature generation and action anticipation called feature mapping RNN .
Our novel RNN architecture builds upon three effective principles of machine
learning, namely parameter sharing, Radial Basis Function kernels and adversarial training. Using only some of the earliest frames of a video, the feature mapping RNN is able to generate future features with a fraction of the parameters
needed in traditional RNN. By feeding these future features into a simple multilayer perceptron facilitated with an RBF kernel layer, we are able to accurately
predict the action in the video.
In our experiments, we obtain 18% improvement on JHMDB-21 dataset, 6% on
UCF101-24 and 13% improvement on UT-Interaction datasets over prior stateof-the-art for action anticipation