Abstract

We present a vision based, adaptive, decision theoretic model of human facial displays in interactions. The model is a partially observ- able Markov decision process, or POMDP. A POMDP is a stochastic planner used by an agent to relate its actions and utility function to its observations and to other context. Video observations are integrated into the POMDP using a dynamic Bayesian network that creates spa- tial and temporal abstractions of the input sequences. The parameters of the model are learned from training data using an a-posteriori con- strained optimization technique based on the expectation-maximization algorithm. The training does not require facial display labels on the training data. The learning process discovers clusters of facial display se- quences and their relationship to the context automatically. This avoids the need for human intervention in training data collection, and allows the models to be used without modi?cation for facial display learning in any context without prior knowledge of the type of behaviors to be used. We present an experimental paradigm in which we record two humans playing a game, and learn the POMDP model of their behaviours. The learned model correctly predicts human actions during a simple cooper- ative card game based, in part, on their facial displays.