Abstract
This paper presents a framework for normbased capacity control with respect to a WeightNormalized Residual Neural Networks (ResNets).
We first formulate the representation of each residual block. For the regression problem, we analyze the Rademacher Complexity of the ResNets family and establish a tighter generalization upper bound for Weight-Normalized ResNets. Using the ?p,q-norm weight normalization in which
1/p+1/q > 1, we discuss the properties of a widthindependent capacity control, which only relies on
the depth according to a square root term. Several
comparisons suggest that our result is tighter than
previous work. Parallel results for Deep Neural
Networks (DNN) and Convolutional Neural Networks (CNN) are included by introducing the ?p,qnorm weight normalization for DNN and the ?p,qnorm kernel normalization for CNN. Numerical experiments also verify that ResNet structures contribute to better generalization properties