Stabilizing Transformer Training Through Consensus
Standard attention-based transformers are known to exhibit instability under learning rate overspecification during training, particularly at high learning rates. While various methods have been proposed to improve resilience to such overspecification by modifying the optimization procedure, fundamental architectural innovations to this end remain underexplored. In this work, we illustrate that the consensus mechanism, a drop-in replacement for attention, stabilizes transformer training across a wider effective range of learning rates. We formulate consensus as a graphical model and provide extensive empirical analysis demonstrating improved stability across learning rate sweeps on text, DNA, and protein modalities. We further propose a hybrid consensus-attention framework that preserves performance while improving stability. We provide theoretical analysis characterizing the properties of consensus.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformers
Transformers have achieved great success in machine learning applications. Normalization techniques, such as Layer Normalization (LayerNorm, LN) and Root Mean Square Normalization (RMSNorm), play a critical role in accel…
B-PL-PINN: Stabilizing PINN Training with Bayesian Pseudo Labeling
Training physics-informed neural networks (PINNs) for forward problems often suffers from severe convergence issues, hindering the propagation of information from regions where the desired solution is well-defined. Haits…
Resilient Model-Free Asymmetric Bipartite Consensus for Nonlinear Multi-Agent Systems against DoS Attacks
In this letter, we study an unified resilient asymmetric bipartite consensus (URABC) problem for nonlinear multi-agent systems with both cooperative and antagonistic interactions under denial-of-service (DoS) attacks. We…
SPAGETI: Stabilizing Phylogenetic Assessment with Gene Evolutionary Tree Indices
The standard approach to estimate species trees is to align a selected set of genes, concatenate the alignments and then estimate a consensus tree. However, individual genes contain differing levels of evolutionary infor…
ProCon: Projection-Consistency Memory for Training-Free Anomaly Detection
Memory-based anomaly detection is attractive because it localizes defects from normal images without training a decoder or synthesizing pseudo anomalies. However, most memory methods still use the memory bank as a neares…
Anomaly Detection