Abstract

We describe a method for automatically associating image patches from frames of a movie shot into object-level groups. The method employs both the appearance and motion of the patches. There are two areas of innovation: first, affine invariant regions are used to repair short gaps in individual tracks and also to join sets of tracks across occlusions (where many tracks are lost simultaneously); second, a robust affine factoriza- tion method is developed which is able to cope with motion degeneracy. This factorization is used to associate tracks into object-level groups. The outcome is that separate parts of an object that are never visible simultaneously in a single frame are associated together. For example, the front and back of a car, or the front and side of a face. In turn this enables object-level matching and recognition throughout a video. We illustrate the method for a number of shots from the feature film ‘Groundhog Day’.