Abstract. We introduce a saliency-based distortion layer for convolutional neural networks that helps to improve the spatial sampling of
input data for a given task. Our differentiable layer can be added as a
preprocessing block to existing task networks and trained altogether in
an end-to-end fashion. The effect of the layer is to efficiently estimate
how to sample from the original data in order to boost task performance.
For example, for an image classification task in which the original data
might range in size up to several megapixels, but where the desired input
images to the task network are much smaller, our layer learns how best to
sample from the underlying high resolution data in a manner which preserves task-relevant information better than uniform downsampling. This
has the effect of creating distorted, caricature-like intermediate images,
in which idiosyncratic elements of the image that improve task performance are zoomed and exaggerated. Unlike alternative approaches such
as spatial transformer networks, our proposed layer is inspired by image
saliency, computed efficiently from uniformly downsampled data, and degrades gracefully to a uniform sampling strategy under uncertainty. We
apply our layer to improve existing networks for the tasks of human gaze
estimation and fine-grained object classification. Code for our method is
available in: http://github.com/recasens/Saliency-Sampler