Abstract

We develop a video understanding system for scene elements, such as bus stops, crosswalks, and intersections, that are characterized more by qualitative activities and geometry than by intrinsic appearance. The domain models for scene elements are not learned from a corpus of video, but instead, naturally elicited by humans, and represented as prob- abilistic logic rules within a Markov Logic Network framework. Human elicited models, however, represent ob ject interactions as they occur in the 3D world rather than describing their appearance pro jection in some specific 2D image plane. We bridge this gap by recovering qualitative scene geometry to analyze ob ject interactions in the 3D world and then reasoning about scene geometry, occlusions and common sense domain knowledge using a set of meta-rules. The effectiveness of this approach is demonstrated on a set of videos of public spaces.