Abstract

Large-scale action recognition and video categorizationare important problems in computer vision. To address these problems, we propose a novel objectand scene-basedsemantic fusion network and representation. Our semantic fusion network combines three streams of information us-ing a three-layer neural network: (i) frame-based low-level CNN features, (ii) object features from a state-of-the-artlarge-scale CNN object-detector trained to recognize 20Kclasses, and (iii) scene features from a state-of-the-art CNNscene-detector trained to recognize 205 scenes. The trained network achieves improvements in supervised activity and video categorization in two complex large-scale datasets ActivityNet and FCVID, respectively. Further, by examining and back propagating information through the fusion net-work, semantic relationships (correlations) between video classes and objects/scenes can be discovered. These video class-object/video class-scene relationships can in turn be used as semantic representation for the video classes themselves. We illustrate effectiveness of this semantic representation through experiments on zero-shot action/video classification and clustering.