资源论文Theoretical Investigation of Generalization Bound for Residual Networks

Theoretical Investigation of Generalization Bound for Residual Networks

2019-09-30 | |  55 |   28 |   0
Abstract This paper presents a framework for normbased capacity control with respect to a WeightNormalized Residual Neural Networks (ResNets). We first formulate the representation of each residual block. For the regression problem, we analyze the Rademacher Complexity of the ResNets family and establish a tighter generalization upper bound for Weight-Normalized ResNets. Using the ?p,q-norm weight normalization in which 1/p+1/q > 1, we discuss the properties of a widthindependent capacity control, which only relies on the depth according to a square root term. Several comparisons suggest that our result is tighter than previous work. Parallel results for Deep Neural Networks (DNN) and Convolutional Neural Networks (CNN) are included by introducing the ?p,qnorm weight normalization for DNN and the ?p,qnorm kernel normalization for CNN. Numerical experiments also verify that ResNet structures contribute to better generalization properties

上一篇:Success Prediction on Crowdfunding with Multimodal Deep Learning

下一篇:Tree Sampling Divergence: An Information-Theoretic Metric for Hierarchical Graph Clustering

用户评价
全部评价

热门资源

  • Learning to Predi...

    Much of model-based reinforcement learning invo...

  • Stratified Strate...

    In this paper we introduce Stratified Strategy ...

  • The Variational S...

    Unlike traditional images which do not offer in...

  • Learning to learn...

    The move from hand-designed features to learned...

  • A Mathematical Mo...

    Direct democracy, where each voter casts one vo...