Abstract

We propose a novel discriminative model for semanticlabeling in videos by incorporating a prior to model boththe shape and temporal dependencies of an object in video.A typical approach for this task is the conditional randomfield (CRF), which can model local interactions among ad-jacent regions in a video frame. Recent work [16, 14]has shown how to incorporate a shape prior into a CRFfor improving labeling performance, but it may be difficultto model temporal dependencies present in video by usingthis prior. The conditional restricted Boltzmann machine(CRBM) can model both shape and temporal dependencies,and has been used to learn walking styles from motion-capture data. In this work, we incorporate a CRBM priorinto a CRF framework and present a new state-of-the-art model for the task of semantic labeling in videos. In particular, we explore the task of labeling parts of complex face scenes from videos in the YouTube Faces Database (YFDB).Our combined model outperforms competitive baselines both qualitatively and quantitatively.