Abstract This paper discusses the problem of recognizing interaction-level human activities from a ?rst-person viewpoint. The goal is to enable an observer (e.g., a robot or a wearable camera) to understand ‘what activity others are performing to it’ from continuous video inputs. These include friendly interactions such as ‘a person hugging the observer’ as well as hostile interactions like ‘punching the observer’ or ‘throwing objects to the observer’, whose videos involve a large amount of camera ego-motion caused by physical interactions. The paper investigates multichannel kernels to integrate global and local motion information, and presents a new activity learning/recognition methodology that explicitly considers temporal structures displayed in ?rst-person activity videos. In our experiments, we not only show classi?cation results with segmented videos, but also con?rm that our new approach is able to detect activities from continuous videos reliably.