Forgetting Memories and their Attractiveness
We study numerically the memory which forgets, introduced in 1986 by Parisi by bounding the synaptic strength, with a mechanism which avoid confusion, allows to remember the pattern learned more recently and has a physiologically very well defined meaning. We analyze a number of features of the learning at finite number of neurons and finite number of patterns. We discuss how the system behaves in the large but finite N limit. We analyze the basin of attraction of the patterns that have been learned, and we show that it is exponentially small in the age of the pattern. This is a clearly non physiological feature of the model.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Memorisation and forgetting in a learning Hopfield neural network: bifurcation mechanisms, attractors and basins
Despite explosive expansion of artificial intelligence based on artificial neural networks (ANNs), these are employed as "black boxes'', as it is unclear how, during learning, they form memories or develop unwanted featu…
Forgetting leads to chaos in attractor networks
Attractor networks are an influential theory for memory storage in brain systems. This theory has recently been challenged by the observation of strong temporal variability in neuronal recordings during memory tasks. In …
RetrievalUse it or Lose it: Selective Memory and Forgetting in a Perpetual Learning Machine
In a recent article we described a new type of deep neural network - a Perpetual Learning Machine (PLM) - which is capable of learning 'on the fly' like a brain by existing in a state of Perpetual Stochastic Gradient Des…
F-MALLOC: Feed-forward Memory Allocation for Continual Learning in Neural Machine Translation
In the evolving landscape of Neural Machine Translation (NMT), the pretrain-then-finetune paradigm has yielded impressive results. However, the persistent challenge of Catastrophic Forgetting (CF) remains a hurdle. While…
Continual LearningMachine TranslationNMTTranslationInterweaving Memories of a Siamese Large Language Model
Parameter-efficient fine-tuning (PEFT) methods optimize large language models (LLMs) by modifying or introducing a small number of parameters to enhance alignment with downstream tasks. However, they can result in catast…
Language ModelingLanguage ModellingLarge Language Modelparameter-efficient fine-tuning+1