Deep equilibrium networks are sensitive to initialization statistics
Deep equilibrium networks (DEQs) are a promising way to construct models which trade off memory for compute. However, theoretical understanding of these models is still lacking compared to traditional networks, in part because of the repeated application of a single set of weights. We show that DEQs are sensitive to the higher order statistics of the matrix families from which they are initialized. In particular, initializing with orthogonal or symmetric matrices allows for greater stability in training. This gives us a practical prescription for initializations which allow for training with a broader range of initial weight scales.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Proximity Variational Inference
Variational inference is a powerful approach for approximate posterior inference. However, it is sensitive to initialization and can be subject to poor local optima. In this paper, we develop proximity variational infere…
Variational InferenceOptimized Weight Initialization on the Stiefel Manifold for Deep ReLU Neural Networks
Stable and efficient training of ReLU networks with large depth is highly sensitive to weight initialization. Improper initialization can cause permanent neuron inactivation dying ReLU and exacerbate gradient instability…
Subcritical Signal Propagation at Initialization in Normalization-Free Transformers
We study signal propagation at initialization in transformers through the averaged partial Jacobian norm (APJN), a measure of gradient amplification across layers. We extend APJN analysis to transformers with bidirection…
Fictitious Play with Maximin Initialization
Fictitious play has recently emerged as the most accurate scalable algorithm for approximating Nash equilibrium strategies in multiplayer games. We show that the degree of equilibrium approximation error of fictitious pl…
Settling: Equilibrium Inference for Non-Convex Validity Sets
Many learning systems return a single point estimate even when admissible outputs form disconnected or non-convex sets. Under squared loss, an ambiguous conditional distribution can therefore have a Bayes-optimal conditi…