Abstract

This work aims to provide understandings on the remarkable success of deep convolutional neural networks (CNNs) by theoretically analyzing their generalization performance and establishing optimization guarantees for gradient descent based training algorithms. Specifically, for a CNN model consisting of l convolutional layers and one fully connected layer, we provepthat its generalization error is bounded by where denotes freedom degree of the network parameQl ters and encapsulates architecture parameters including the kernel size ki , stride si , pooling p and parameter magnitude bi . To our best knowledge, this is the first generalization bound that Ql+1 only depends on , tighter than existing ones that all involve an exponential term  like . Besides, we prove that for an ar bitrary gradient descent algorithm, the computed approximate stationary point by minimizing empirical risk is also an approximate stationary poi to the population risk. This well explains why gra dient descent training algorithms usually perform sufficiently well in practice. Furthermore, we prove the one-to-one correspondence and convergence guarantees for the non-degenerate stationary points between the empirical and population risks. It implies that the computed local minimum for the empirical risk is also close to a local mi imum for the population risk, thus ensuring the good generalization performance of CNNs.