Abstract Detecting abnormal activities from sensor readings is an important research problem in activity recognition. A number of different algorithms have been proposed in the past to tackle this problem. Many of the previous state-based approaches suffer from the problem of failing to decide the appropriate number of states, which are diffificult to fifind through a trial-and-error approach, in real-world applications. In this paper, we propose an accurate and flflexible framework for abnormal activity recognition from sensor readings that involves less human tuning of model parameters. Our approach fifirst applies a Hierarchical Dirichlet Process Hidden Markov Model (HDP-HMM), which supports an infifinite number of states, to automatically fifind an appropriate number of states. We incorporate a Fisher Kernel into the One-Class Support Vector Machine (OCSVM) model to fifilter out the activities that are likely to be normal. Finally, we derive an abnormal activity model from the normal activity models to reduce false positive rate in an unsupervised manner. Our main contribution is that our proposed HDP-HMM models can decide the appropriate number of states automatically, and that by incorporating a Fisher Kernel into the OCSVM model, we can combine the advantages from generative model and discriminative model. We demonstrate the effectiveness of our approach by using several real-world datasets to test our algorithm’s performance