BranchNorm: Robustly Scaling Extremely Deep Transformers

2023-05-04 · Yijin Liu, Xianfeng Zeng, Fandong Meng, Jie zhou

Recently, DeepNorm scales Transformers into extremely deep (i.e., 1000 layers) and reveals the promising potential of deep scaling. To stabilize the training of deep models, DeepNorm (Wang et al., 2022) attempts to constrain the model update to a constant value. Although applying such a constraint can benefit the early stage of model training, it may lead to undertrained models during the whole training procedure. In this paper, we propose BranchNorm, which dynamically rescales the non-residual branch of Transformer in accordance with the training period. BranchNorm not only theoretically stabilizes the training with smooth gradient norms at the early stage, but also encourages better convergence in the subsequent training stage. Experiment results on multiple translation tasks demonstrate that BranchNorm achieves a better trade-off between training stability and converge performance.

📄 PDF Abstract BibTeX arXiv:2305.02790

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Adam 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Multi-Head Attention 설명 없음

DeepNet: Scaling Transformers to 1,000 Layers

2022-03-01 · Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang 외

In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers. Specifically, we introduce a new normalization function (DeepNorm) to modify the residual connection in Transformer, accom…

Translation

Auto-scaling Vision Transformers without Training

2022-02-24 · ICLR 2022 4 · Wuyang Chen, Wei Huang, Xianzhi Du, Xiaodan Song 외

This work targets automated designing and scaling of Vision Transformers (ViTs). The motivation comes from two pain spots: 1) the lack of efficient and principled methods for designing and scaling ViTs; 2) the tremendous…

GPU

Multilingual estimation of political-party positioning: From label aggregation to long-input Transformers

2023-10-19 · Dmitry Nikolaev, Tanise Ceron, Sebastian Padó

Scaling analysis is a technique in computational political science that assigns a political actor (e.g. politician or party) a score on a predefined scale based on a (typically long) body of text (e.g. a parliamentary sp…

Scaling Transformers for Low-Bitrate High-Quality Speech Coding

2024-11-29 · Julian D Parker, Anton Smirnov, Jordi Pons, CJ Carr 외

The tokenization of speech with neural audio codec models is a vital part of modern AI pipelines for the generation or understanding of speech, alone or in a multimodal context. Traditionally such tokenization models hav…

Quantization

Grokking of Hierarchical Structure in Vanilla Transformers

2023-05-30 · Shikhar Murty, Pratyusha Sharma, Jacob Andreas, Christopher D. Manning

For humans, language production and comprehension is sensitive to the hierarchical structure of sentences. In natural language processing, past work has questioned how effectively neural sequence models like transformers…