Exploring the Temperature-Dependent Phase Transition in Modern Hopfield Networks
The recent discovery of a connection between Transformers and Modern Hopfield Networks (MHNs) has reignited the study of neural networks from a physical energy-based perspective. This paper focuses on the pivotal effect of the inverse temperature hyperparameter $\beta$ on the distribution of energy minima of the MHN. To achieve this, the distribution of energy minima is tracked in a simplified MHN in which equidistant normalised patterns are stored. This network demonstrates a phase transition at a critical temperature $\beta_{\text{c}}$, from a single global attractor towards highly pattern specific minima as $\beta$ is increased. Importantly, the dynamics are not solely governed by the hyperparameter $\beta$ but are instead determined by an effective inverse temperature $\beta_{\text{eff}}$ which also depends on the distribution and size of the stored patterns. Recognizing the role of hyperparameters in the MHN could, in the future, aid researchers in the domain of Transformers to optimise their initial choices, potentially reducing the necessity for time and energy expensive hyperparameter fine-tuning.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Exploring Neurofunctional Phase Transition Patterns in Autism Spectrum Disorder: A Thermodynamics Parameters Analysis Approach
Designing network parameters that can effectively represent complex networks is of significant importance for the analysis of time-varying complex networks. This paper introduces a novel thermodynamic framework for analy…
Unsupervised machine learning approaches to the $q$-state Potts model
In this paper with study phase transitions of the $q$-state Potts model, through a number of unsupervised machine learning techniques, namely Principal Component Analysis (PCA), $k$-means clustering, Uniform Manifold App…
BIG-bench Machine LearningClusteringTopological Data AnalysisPlanted vertex cover problem on regular random graphs and nonmonotonic temperature-dependence in the supercooled region
We introduce a planted vertex cover problem on regular random graphs and study it by the cavity method of statistical mechanics. Different from conventional Ising models, the equilibrium ferromagnetic phase transition of…
On the behavior of random RNA secondary structures near the glass transition
RNA forms elaborate secondary structures through intramolecular base pairing. These structures perform critical biological functions within each cell. Due to the availability of a polynomial algorithm to calculate the pa…
Attention to Order: Transformers Discover Phase Transitions via Learnability
Phase transitions mark qualitative reorganizations of collective behavior, yet identifying their boundaries remains challenging whenever analytic solutions are absent and conventional simulations fail. Here we introduce …
Self-Supervised Learning