paper-with-me

홈 › Papers

Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models

2022-02-09 · Beren Millidge, Tommaso Salvatori, Yuhang Song, Thomas Lukasiewicz, Rafal Bogacz

A large number of neural network models of associative memory have been proposed in the literature. These include the classical Hopfield networks (HNs), sparse distributed memories (SDMs), and more recently the modern continuous Hopfield networks (MCHNs), which possesses close links with self-attention in machine learning. In this paper, we propose a general framework for understanding the operation of such memory networks as a sequence of three operations: similarity, separation, and projection. We derive all these memory models as instances of our general framework with differing similarity and separation functions. We extend the mathematical framework of Krotov et al (2020) to express general associative memory models using neural network dynamics with only second-order interactions between neurons, and derive a general energy function that is a Lyapunov function of the dynamics. Finally, using our framework, we empirically investigate the capacity of using different similarity functions for these associative memory models, beyond the dot product similarity measure, and demonstrate empirically that Euclidean or Manhattan distance similarity metrics perform substantially better in practice on many tasks, enabling a more robust retrieval and higher memory capacity than existing models.

📄 PDF Abstract BibTeX arXiv:2202.04557

Code (1)

BerenMillidge/Theory_Associative_Memory 공식 구현 pytorch

Tasks

Retrieval

Similar Papers 제목 키워드 기반

Hopfield-Fenchel-Young Networks: A Unified Framework for Associative Memory Retrieval

2024-11-13 · Saul Santos, Vlad Niculae, Daniel McNamee, André F. T. Martins

Associative memory models, such as Hopfield networks and their modern variants, have garnered renewed interest due to advancements in memory capacity and connections with self-attention in transformers. In this work, we …

Image RetrievalMultiple Instance LearningRetrieval

HOMOE: A Memory-Based and Composition-Aware Framework for Zero-Shot Learning with Hopfield Network and Soft Mixture of Experts

2023-11-23 · Do Huu Dat, Po Yuan Mao, Tien Hoang Nguyen, Wray Buntine 외

Compositional Zero-Shot Learning (CZSL) has emerged as an essential paradigm in machine learning, aiming to overcome the constraints of traditional zero-shot learning by incorporating compositional thinking into its meth…

Compositional Zero-Shot LearningMixture-of-ExpertsZero-Shot Learning

Improving Few- and Zero-Shot Reaction Template Prediction Using Modern Hopfield Networks

2022-01-15 · Journal of Chemical Information and Modeling 2022 1 · Philipp Seidl, Philipp Renz, Natalia Dyubankova, Paulo Neves 외

Finding synthesis routes for molecules of interest is essential in the discovery of new drugs and materials. To find such routes, computer-assisted synthesis planning (CASP) methods are employed, which rely on a single-s…

RetrosynthesisSingle-step retrosynthesis

Sparse and Structured Hopfield Networks

2024-02-21 · Saul Santos, Vlad Niculae, Daniel McNamee, Andre F. T. Martins

Modern Hopfield networks have enjoyed recent interest due to their connection to attention in transformers. Our paper provides a unified framework for sparse Hopfield networks by establishing a link with Fenchel-Young lo…

Multiple Instance LearningRetrieval

Semantic Iterative Reconstruction: One-Shot Universal Anomaly Detection

2026-03-24 · Ning Zhu arxiv

Unsupervised medical anomaly detection is severely limited by the scarcity of normal training samples. Existing methods typically train dedicated models for each dataset or disease, requiring hundreds of normal images pe…

Anomaly Detection