Abstract

The collective activity of a group of persons is more than a mere sum of individual person actions, since interactions and the context of the over- all group behavior have crucial influence. Consequently, the current standard paradigm for group activity recognition is to model the spatiotemporal pattern of individual person bounding boxes and their interactions. Despite this trend towards increasingly global representations, activities are often defined by semi- local characteristics and their interrelation between different persons. For captur- ing the large visual variability with small semi-local parts, a large number of them are required, thus rendering manual annotation infeasible. To automatically learn activity constituents that are meaningful for the collective activity, we sample lo- cal parts and group related ones not merely based on visual similarity but based on the function they ful fill on a set of validation images. Then max-margin mul- tiple instance learning is employed to jointly i) remove clutter from these groups and focus on only the relevant samples, ii) learn the activity constituents, and iii) train the multi-class activity classi fier. Experiments on standard activity bench- mark sets show the advantage of this joint procedure and demonstrate the benefit of functionally grouped latent activity constituents for group activity recognition.