Abstract

We present a coherent, discriminative framework for simul- taneously tracking multiple people and estimating their collective ac- tivities. Instead of treating the two problems separately, our model is grounded in the intuition that a strong correlation exists between a per- son’s motion, their activity, and the motion and activities of other nearby people. Instead of directly linking the solutions to these two problems, we introduce a hierarchy of activity types that creates a natural pro- gression that leads from a specific person’s motion to the activity of the group as a whole. Our model is capable of jointly tracking multiple peo- ple, recognizing individual activities (atomic activities ), the interactions between pairs of people (interaction activities ), and finally the behavior of groups of people (col lective activities ). We also propose an algorithm for solving this otherwise intractable joint inference problem by combin- ing belief propagation with a version of the branch and bound algorithm equipped with integer programming. Experimental results on challenging video datasets demonstrate our theoretical claims and indicate that our model achieves the best collective activity classification results to date.