Transformers learn through gradual rank increase
We identify incremental learning dynamics in transformers, where the difference between trained and initial weights progressively increases in rank. We rigorously prove this occurs under the simplifying assumptions of diagonal weight matrices and small initialization. Our experiments support the theory and also show that phenomenon can occur in practice without the simplifying assumptions.
Code (0)
등록된 구현이 없습니다.
Tasks
Incremental LearningSimilar Papers 제목 키워드 기반
InRank: Incremental Low-Rank Learning
The theory of greedy low-rank learning (GLRL) aims to explain the impressive generalization capabilities of deep learning. It proves that stochastic gradient-based training implicitly regularizes neural networks towards …
Computational EfficiencyDissecting Lottery Ticket Transformers: Structural and Behavioral Study of Sparse Neural Machine Translation
Recent work on the lottery ticket hypothesis has produced highly sparse Transformers for NMT while maintaining BLEU. However, it is unclear how such pruning techniques affect a model's learned representations. By probing…
Machine TranslationNMTTranslationCoarse-to-Fine Contrastive Learning on Graphs
Inspired by the impressive success of contrastive learning (CL), a variety of graph augmentation strategies have been employed to learn node representations in a self-supervised manner. Existing methods construct the con…
Contrastive LearningLearning-To-RankDynamic Token-Pass Transformers for Semantic Segmentation
Vision transformers (ViT) usually extract features via forwarding all the tokens in the self-attention layers from top to toe. In this paper, we introduce dynamic token-pass vision transformers (DoViT) for semantic segme…
SegmentationSemantic SegmentationOn the Role of Attention Masks and LayerNorm in Transformers
Self-attention is the key mechanism of transformers, which are the essential building blocks of modern foundation models. Recent studies have shown that pure self-attention suffers from an increasing degree of rank colla…