Beyond Folklore: A Scaling Calculus for the Design and Initialization of ReLU Networks
We propose a system for calculating a "scaling constant" for layers and weights of neural networks. We relate this scaling constant to two important quantities that relate to the optimizability of neural networks, and argue that a network that is "preconditioned" via scaling, in the sense that all weights have the same scaling constant, will be easier to train. This scaling calculus results in a number of consequences, among them the fact that the geometric mean of the fan-in and fan-out, rather than the fan-in, fan-out, or arithmetic mean, should be used for the initialization of the variance of weights in a neural network. Our system allows for the off-line design & engineering of ReLU neural networks, potentially replacing blind experimentation.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Reconciling Lambek's restriction, cut-elimination, and substitution in the presence of exponential modalities
The Lambek calculus can be considered as a version of non-commutative intuitionistic linear logic. One of the interesting features of the Lambek calculus is the so-called "Lambek's restriction," that is, the antecedent o…
Myths and Legends in High-Performance Computing
In this thought-provoking article, we discuss certain myths and legends that are folklore among members of the high-performance computing community. We gathered these myths from conversations at conferences and meetings,…
ArticlesVocal Bursts Intensity PredictionA Systematic Study and Analysis of Bengali Folklore with Natural Language Processing Systems
Folklore, a solid branch of folk literature, is the hallmark of any nation or any society. Such as oral tradition; as proverbs or jokes, it also includes material culture as well as traditional folk beliefs, and various …
Cultural Vocal Bursts Intensity PredictionScaling Laws for the Principled Design, Initialization, and Preconditioning of ReLU Networks
Abstract In this work, we describe a set of rules for the design and initialization of well-conditioned neural networks, guided by the goal of naturally balancing the diagonal blocks of the Hessian at the start of traini…
Scale Normalization
One of the difficulties of training deep neural networks is caused by improper scaling between layers. Scaling issues introduce exploding / gradient problems, and have typically been addressed by careful scale-preserving…