Abstract Automated detection of anomalous events plays an important role in video surveillance systems in practice. This task, however, requires to deal with three challenging problems of the lack of annotated training data, the inexact description of what to be “abnormal” and the expensive feature engineering procedure. Most anomaly detection systems are only able to satisfy some of these challenges. In this work, we propose a deep abnormality detection system to handle all of them simultaneously. Deep abnormality detection is a deep generative network that is an unsupervised probabilistic framework to model the normality and learn feature representation automatically. Furthermore, unlike other existing methods, our system can detect abnormality at multiple levels and be used as a powerful tool for video analysis and scene understanding.