资源分类
》 资源 》论文》PRINCIPLED WEIGHT INITIALIZATION FOR HYPERNETW ORKS

PRINCIPLED WEIGHT INITIALIZATION FOR HYPERNETW ORKS

2020-01-02 | |  80 |   58 |   0

Abstract

Hypernetworks are meta neural networks that generate weights for a main neural network in an end-to-end differentiable manner. Despite extensive applications ranging from multi-task learning to Bayesian deep learning, the problem of optimizing hypernetworks has not been studied to date. We observe that classical weight initialization methods like Glorot & Bengio (2010) and He et al. (2015), when applied directly on a hypernet, fail to produce weights for the mainnet in the correct scale. We develop principled techniques for weight initialization in hypernets, and show that they lead to more stable mainnet weights, lower training loss, and faster convergence.

上一篇：BLACK -B OX ADVERSARIAL ATTACK WITH TRANS -FERABLE MODEL -BASED EMBEDDING

下一篇：UNDERSTANDING AND ROBUSTIFYINGD IFFERENTIABLE ARCHITECTURE SEARCH

用户评价
全部评价

热门资源

  • Stratified Strate...

    In this paper we introduce Stratified Strategy ...

  • The Variational S...

    Unlike traditional images which do not offer in...

  • Learning to learn...

    The move from hand-designed features to learned...

  • A Mathematical Mo...

    Direct democracy, where each voter casts one vo...

  • Learning to Predi...

    Much of model-based reinforcement learning invo...