Abstract
Even though probabilistic treatments of neural networks
have a long history, they have not found widespread use in
practice. Sampling approaches are often too slow already
for simple networks. The size of the inputs and the depth
of typical CNN architectures in computer vision only compound this problem. Uncertainty in neural networks has
thus been largely ignored in practice, despite the fact that it
may provide important information about the reliability of
predictions and the inner workings of the network. In this
paper, we introduce two lightweight approaches to making
supervised learning with probabilistic deep networks practical: First, we suggest probabilistic output layers for classification and regression that require only minimal changes
to existing networks. Second, we employ assumed density
filtering and show that activation uncertainties can be propagated in a practical fashion through the entire network,
again with minor changes. Both probabilistic networks retain the predictive power of the deterministic counterpart,
but yield uncertainties that correlate well with the empirical error induced by their predictions. Moreover, the robustness to adversarial examples is significantly increased