资源分类
》 资源 》论文》Path-Normalized Optimization of Recurrent Neural Networks with ReLU Activations

Path-Normalized Optimization of Recurrent Neural Networks with ReLU Activations

2020-02-07 | |  71 |   44 |   0

Abstract 

We investigate the parameter-space geometry of recurrent neural networks (RNNs), and develop an adaptation of path-SGD optimization method, attuned to this geometry, that can learn plain RNNs with ReLU activations. On several datasets that require capturing long-term dependency structure, we show that path-SGD can significantly improve trainability of ReLU RNNs compared to RNNs trained with SGD, even with various recently suggested initialization schemes.

上一篇：Unsupervised Learning of 3D Structure from Images

下一篇：The Parallel Knowledge Gradient Method for Batch Bayesian Optimization

用户评价
全部评价

热门资源

  • The Variational S...

    Unlike traditional images which do not offer in...

  • Learning to Predi...

    Much of model-based reinforcement learning invo...

  • Stratified Strate...

    In this paper we introduce Stratified Strategy ...

  • A Mathematical Mo...

    Direct democracy, where each voter casts one vo...

  • Rating-Boosted La...

    The performance of a recommendation system reli...