Abstract

Graph-theoretical methods have successfully provided semantic and structural interpretations of images and videos. A recent paper introduced a pattern-theoretic approach that allows construction of flexible graphs for representing interactions of actors with objects and inference is accomplished by an efficient annealing algorithm. Actions and objects are termed generators and their interactions are termed bonds; together they form high-probability configurations, or interpretations, of observed scenes. This work and other structural methods have generally been limited to analyzing short videos involving isolated actions.Here we provide an extension that uses additional temporalbonds across individual actions to enable semantic inter-pretations of longer videos. Longer temporal connectionsimprove scene interpretations as they help discard (tem-porally) local solutions in favor of globally superior ones.Using this extension, we demonstrate improvements in un-derstanding longer videos, compared to individual inter-pretations of non-overlapping time segments. We verifiedthe success of our approach by generating interpretationsfor more than 700 video segments from the YouCook dataset, with intricate videos that exhibit cluttered background,scenarios of occlusion, viewpoint variations and changing conditions of illumination. Interpretations for long video segments were able to yield performance increases of about 70% and, in addition, proved to be more robust to different severe scenarios of classification errors.