Abstract

 Variational inference with -divergences has been widely used in modern probabilistic machine learning. Compared to Kullback-Leibler (KL) divergence, a major advantage of using -divergences (with positive values) is their mass-covering property. However, estimating and optimizing -divergences require to use importance sampling, which may have large or infinite variance due to heavy tails of importance weights. In this paper, we propose a new class of tail-adaptive f divergences that adaptively change the convex function f with the tail distribution of the importance weights, in a way that theoretically guarantees finite moments, while simultaneously achieving mass-covering properties. We test our method on Bayesian neural networks, and apply it to improve a recent soft actor-critic (SAC) algorithm (Haarnoja et al., 2018) in deep reinforcement learning. Our results show that our approach yields significant advantages compared with existing methods based on classical KL and -divergences.