Improving Transformer Optimization Through Better Initialization
The Transformer architecture has achieved considerable success in areas such as language modeling and machine translation. The key component of the Transformer is the attention layer that enables the model to focus on important regions within the input sequence. Gradient optimization with attention layers can be notoriously difficult requiring tricks such as learning rate warmup to prevent divergence. As Transformer models are becoming larger and more expensive to train, recent research has focused on understanding and improving optimization in these models. In this work our contributions are two-fold. We first investigate and empirically validate the source of optimization problems in encoder-decoder Transformer architecture.We then propose a new weight initialization scheme with theoretical justification, which enables training without warmup or layer normalization. Empirical results on public machine translation benchmarks show that our approach achieves leading accuracy, allowing to train deep Transformer models with 200 layers without difficulty. Full code for this work will be released with the final version of this draft.
Code (1)
Tasks
DecoderLanguage ModelingLanguage ModellingMachine TranslationTranslationSimilar Papers 제목 키워드 기반
Improving Transformer Optimization Through Better Initialization
The Transformer architecture has achieved considerable success in areas such as language modeling and machine translation. The key component of the Transformer is the attention layer that enables the model to focus on im…
DecoderLanguage ModelingLanguage ModellingMachine Translation+1Subcritical Signal Propagation at Initialization in Normalization-Free Transformers
We study signal propagation at initialization in transformers through the averaged partial Jacobian norm (APJN), a measure of gradient amplification across layers. We extend APJN analysis to transformers with bidirection…
Towards Theoretically Inspired Neural Initialization Optimization
Automated machine learning has been widely explored to reduce human efforts in designing neural architectures and looking for proper hyperparameters. In the domain of neural initialization, however, similar automated tec…
Lipschitz Constrained Parameter Initialization for Deep Transformers
The Transformer translation model employs residual connection and layer normalization to ease the optimization difficulties caused by its multi-layer encoder/decoder structure. Previous research shows that even with resi…
DecoderTranslationCritical Windows of Complexity Control: When Transformers Decide to Reason or Memorize
Recent work has shown that Transformers' compositional generalization is governed by \emph{complexity control}, initialization scale and weight decay, which steers training toward low-complexity reasoning solutions rathe…