Learning for Single-Shot Confidence Calibration in Deep Neural Networksthrough Stochastic Inferences
Abstract
We propose a generic framework to calibrate accuracy
and confidence of a prediction in deep neural networks
through stochastic inferences. We interpret stochastic regularization using a Bayesian model, and analyze the relation
between predictive uncertainty of networks and variance of
the prediction scores obtained by stochastic inferences for
a single example. Our empirical study shows that the accuracy and the score of a prediction are highly correlated
with the variance of multiple stochastic inferences given by
stochastic depth or dropout. Motivated by this observation,
we design a novel variance-weighted confidence-integrated
loss function that is composed of two cross-entropy loss
terms with respect to ground-truth and uniform distribution, which are balanced by variance of stochastic prediction scores. The proposed loss function enables us to learn
deep neural networks that predict confidence calibrated
scores using a single inference. Our algorithm presents outstanding confidence calibration performance and improves
classification accuracy when combined with two popular
stochastic regularization techniques—stochastic depth and
dropout—in multiple models and datasets; it alleviates
overconfidence issue in deep neural networks significantly
by training networks to achieve prediction accuracy proportional to confidence of prediction.