How to explain grokking
Explanation of grokking (delayed generalization) in learning is given by modeling grokking by the stochastic gradient Langevin dynamics (Brownian motion) and applying the ideas of thermodynamics.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Grokking Explained: A Statistical Phenomenon
Grokking, or delayed generalization, is an intriguing learning phenomenon where test set loss decreases sharply only after a model's training set loss has converged. This challenges conventional understanding of the trai…
Explaining Grokking in Transformers through the Lens of Inductive Bias
We investigate grokking in transformers through the lens of inductive bias: dispositions arising from architecture or optimization that let the network prefer one solution over another. We first show that architectural c…
Omnigrok: Grokking Beyond Algorithmic Data
Grokking, the unusual phenomenon for algorithmic datasets where generalization happens long after overfitting the training data, has remained elusive. We aim to understand grokking by analyzing the loss landscapes of neu…
AttributeRepresentation LearningModel Capacity Determines Grokking through Competing Memorisation and Generalisation Speeds
Existing accounts of grokking explain the phenomena in terms of mechanistic frameworks such as circuit efficiency or lazy-to-rich transitions. However, despite a known dependence between grokking and model size, how mode…
Measuring Sharpness in Grokking
Neural networks sometimes exhibit grokking, a phenomenon where perfect or near-perfect performance is achieved on a validation set well after the same performance has been obtained on the corresponding training set. In t…