Abrupt Learning in Transformers: A Case Study on Matrix Completion
Recent analysis on the training dynamics of Transformers has unveiled an interesting characteristic: the training loss plateaus for a significant number of training steps, and then suddenly (and sharply) drops to near--optimal values. To understand this phenomenon in depth, we formulate the low-rank matrix completion problem as a masked language modeling (MLM) task, and show that it is possible to train a BERT model to solve this task to low error. Furthermore, the loss curve shows a plateau early in training followed by a sudden drop to near-optimal values, despite no changes in the training procedure or hyper-parameters. To gain interpretability insights into this sudden drop, we examine the model's predictions, attention heads, and hidden states before and after this transition. Concretely, we observe that (a) the model transitions from simply copying the masked input to accurately predicting the masked entries; (b) the attention heads transition to interpretable patterns relevant to the task; and (c) the embeddings and hidden states encode information relevant to the problem. We also analyze the training dynamics of individual model components to understand the sudden drop in loss.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingLow-Rank Matrix CompletionMasked Language ModelingMatrix CompletionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Locating and Editing Figure-Ground Organization in Vision Transformers
Vision Transformers must resolve figure-ground organization by choosing between completions driven by local geometric evidence and those favored by global organizational priors, giving rise to a characteristic perceptual…
Algebraic Variety Models for High-Rank Matrix Completion
We consider a generalization of low-rank matrix completion to the case where the data belongs to an algebraic variety, i.e. each data point is a solution to a system of polynomial equations. In this case the original mat…
ClusteringLow-Rank Matrix CompletionMatrix CompletionVocal Bursts Intensity PredictionOn Deterministic Sampling Patterns for Robust Low-Rank Matrix Completion
In this letter, we study the deterministic sampling patterns for the completion of low rank matrix, when corrupted with a sparse noise, also known as robust matrix completion. We extend the recent results on the determin…
Low-Rank Matrix CompletionMatrix CompletionvalidRiemannian Perspective on Matrix Factorization
We study the non-convex matrix factorization approach to matrix completion via Riemannian geometry. Based on an optimization formulation over a Grassmannian manifold, we characterize the landscape based on the notion of …
Matrix CompletionCausal Matrix Completion
Matrix completion is the study of recovering an underlying matrix from a sparse subset of noisy observations. Traditionally, it is assumed that the entries of the matrix are "missing completely at random" (MCAR), i.e., e…
Matrix CompletionRecommendation Systems