Abstract. Despite a lack of theoretical understanding, deep neural networks have achieved unparalleled performance in a wide range of applications. On the other hand, shallow representation learning with component analysis is associated with rich intuition and theory, but smaller
capacity often limits its usefulness. To bridge this gap, we introduce Deep
Component Analysis (DeepCA), an expressive multilayer model formulation that enforces hierarchical structure through constraints on latent
variables in each layer. For inference, we propose a diferentiable optimization algorithm implemented using recurrent Alternating Direction
Neural Networks (ADNNs) that enable parameter learning using standard backpropagation. By interpreting feed-forward networks as singleiteration approximations of inference in our model, we provide both
a novel perspective for understanding them and a practical technique
for constraining predictions with prior knowledge. Experimentally, we
demonstrate performance improvements on a variety of tasks, including
single-image depth prediction with sparse output constraints