Abstract

Generalization performance is a central goal in machine learning, with explicit generalization strategies needed when training over-parametrized models, like large neural networks. There is growing interest in using multiple, potentially auxiliary tasks, as one strategy towards this goal. In this work, we theoretically and empirically analyze one such model, called a supervised auto-encoder: a neural network that jointly predicts targets and inputs (reconstruction). We provide a novel generalization result for linear auto-encoders, proving uniform stability based on the inclusion of the reconstruction error—particularly as an improvement on simplistic regularization such as norms. We then demonstrate empirically that, across an array of architectures with a different number of hidden units and activation functions, the supervised auto-encoder compared to the corresponding standard neural network never harms performance and can improve generalization.