Abstract
RANSAC is an important algorithm in robust optimization and a central building block for many computer vision
applications. In recent years, traditionally hand-crafted
pipelines have been replaced by deep learning pipelines,
which can be trained in an end-to-end fashion. However,
RANSAC has so far not been used as part of such deep
learning pipelines, because its hypothesis selection procedure is non-differentiable. In this work, we present two different ways to overcome this limitation. The most promising
approach is inspired by reinforcement learning, namely to
replace the deterministic hypothesis selection by a probabilistic selection for which we can derive the expected loss
w.r.t. to all learnable parameters. We call this approach
DSAC, the differentiable counterpart of RANSAC. We apply
DSAC to the problem of camera localization, where deep
learning has so far failed to improve on traditional approaches. We demonstrate that by directly minimizing the
expected loss of the output camera poses, robustly estimated
by RANSAC, we achieve an increase in accuracy. In the future, any deep learning pipeline can use DSAC as a robust
optimization component