Abstract
We observed that recent state-of-the-art results on single image human pose estimation were achieved by multistage Convolution Neural Networks (CNN). Notwithstanding the superior performance on static images, the application of these models on videos is not only computationally
intensive, it also suffers from performance degeneration and
flicking. Such suboptimal results are mainly attributed to
the inability of imposing sequential geometric consistency,
handling severe image quality degradation (e.g. motion
blur and occlusion) as well as the inability of capturing the
temporal correlation among video frames. In this paper, we
proposed a novel recurrent network to tackle these problems. We showed that if we were to impose the weight sharing scheme to the multi-stage CNN, it could be re-written
as a Recurrent Neural Network (RNN). This property decouples the relationship among multiple network stages and
results in significantly faster speed in invoking the network
for videos. It also enables the adoption of Long Short-Term
Memory (LSTM) units between video frames. We found such
memory augmented RNN is very effective in imposing geometric consistency among frames. It also well handles input quality degradation in videos while successfully stabilizes the sequential outputs. The experiments showed that
our approach significantly outperformed current state-ofthe-art methods on two large-scale video pose estimation
benchmarks. We also explored the memory cells inside the
LSTM and provided insights on why such mechanism would
benefit the prediction for video-based pose estimations