资源分类
》 资源 》论文》The asymptotic spectrum of the Hessian ofDNN throughout training

The asymptotic spectrum of the Hessian ofDNN throughout training

2020-01-02 | |  66 |   41 |   0

Abstract

The dynamics of DNNs during gradient descent is described by the so-called Neural Tangent Kernel (NTK). In this article, we show that the NTK allows one to gain precise insight into the Hessian of the cost of DNNs. When the NTK is fixed during training, we obtain a full characterization of the asymptotics of the spectrum of the Hessian, at initialization and during training. In the so-called mean-field limit, where the NTK is not fixed during training, we describe the first two moments of the Hessian at initialization.

上一篇：ADVECTIVE NET: AN EULERIAN -L AGRANGIANF LUIDIC RESERVOIR FOR POINT CLOUD PROCESSING

下一篇：NON -AUTOREGRESSIVE DIALOG STATE TRACKING

用户评价
全部评价

热门资源

  • The Variational S...

    Unlike traditional images which do not offer in...

  • Learning to Predi...

    Much of model-based reinforcement learning invo...

  • Stratified Strate...

    In this paper we introduce Stratified Strategy ...

  • A Mathematical Mo...

    Direct democracy, where each voter casts one vo...

  • Rating-Boosted La...

    The performance of a recommendation system reli...