Abstract

Multiple Hypothesis Video Segmentation (MHVS) is a method for the unsupervised photometric segmentation of video se- quences. MHVS segments arbitrarily long video streams by considering only a few frames at a time, and handles the automatic creation, continu- ation and termination of labels with no user initialization or supervision. The process begins by generating several pre-segmentations per frame and enumerating multiple possible tra jectories of pixel regions within a short time window. After assigning each tra jectory a score, we let the tra jectories compete with each other to segment the sequence. We de- termine the solution of this segmentation problem as the MAP labeling of a higher-order random field. This framework allows MHVS to achieve spatial and temporal long-range label consistency while operating in an on-line manner. We test MHVS on several videos of natural scenes with arbitrary camera and ob ject motion.