Attention to Order: Transformers Discover Phase Transitions via Learnability
Phase transitions mark qualitative reorganizations of collective behavior, yet identifying their boundaries remains challenging whenever analytic solutions are absent and conventional simulations fail. Here we introduce learnability as a universal criterion, defined as the ability of a transformer model containing attention mechanism to extract structure from microscopic states. Using self-supervised learning and Monte Carlo generated configurations of the two-dimensional Ising model, we show that ordered phases correspond to enhanced learnability, manifested in both reduced training loss and structured attention patterns, while disordered phases remain resistant to learning. Two unsupervised diagnostics, the sharp jump in training loss and the rise in attention entropy, recover the critical temperature in excellent agreement with the exact value. Our results establish learnability as a data-driven marker of phase transitions and highlight deep parallels between long-range order in condensed matter and the emergence of structure in modern language models.
Code (0)
등록된 구현이 없습니다.
Tasks
Self-Supervised LearningSimilar Papers 제목 키워드 기반
Discovering Phase Transitions with Unsupervised Learning
Unsupervised learning is a discipline of machine learning which aims at discovering patterns in big data sets or classifying the data into several categories without being trained explicitly. We show that unsupervised le…
ClusteringPhase Transitions in Attention: A Bayesian Theory of Copy Head Emergence
Attention is the key mechanism underlying in-context learning in transformers, and attention patterns have been observed empirically to emerge abruptly during training. We present a Bayesian theory of feature learning in…
Unsupervised machine learning of quantum phase transitions using diffusion maps
Experimental quantum simulators have become large and complex enough that discovering new physics from the huge amount of measurement data can be quite challenging, especially when little theoretical understanding of the…
BIG-bench Machine LearningClusteringDimensionality ReductionInterpretable Phase Detection and Classification with Persistent Homology
We apply persistent homology to the task of discovering and characterizing phase transitions, using lattice spin models from statistical physics for working examples. Persistence images provide a useful representation of…
ClassificationGeneral ClassificationregressionPhase Transitions for the Information Bottleneck in Representation Learning
In the Information Bottleneck (IB), when tuning the relative strength between compression and prediction terms, how do the two terms behave, and what's their relationship with the dataset and the learned representation? …
Representation Learning