SNAP: Stopping Catastrophic Forgetting in Hebbian Learning with Sigmoidal Neuronal Adaptive Plasticity
Artificial Neural Networks (ANNs) suffer from catastrophic forgetting, where the learning of new tasks causes the catastrophic forgetting of old tasks. Existing Machine Learning (ML) algorithms, including those using Stochastic Gradient Descent (SGD) and Hebbian Learning typically update their weights linearly with experience i.e., independently of their current strength. This contrasts with biological neurons, which at intermediate strengths are very plastic, but consolidate with Long-Term Potentiation (LTP) once they reach a certain strength. We hypothesize this mechanism might help mitigate catastrophic forgetting. We introduce Sigmoidal Neuronal Adaptive Plasticity (SNAP) an artificial approximation to Long-Term Potentiation for ANNs by having the weights follow a sigmoidal growth behaviour allowing the weights to consolidate and stabilize when they reach sufficiently large or small values. We then compare SNAP to linear weight growth and exponential weight growth and see that SNAP completely prevents the forgetting of previous tasks for Hebbian Learning but not for SGD-base learning.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Differentiable Hebbian Plasticity for Continual Learning
Catastrophic forgetting poses a grand challenge for continual learning systems, which prevents neural networks from protecting old knowledge while learning new tasks sequentially. We propose a Differentiable Hebbian Plas…
Continual LearningPermuted-MNISTSplit-MNISTRepresentation Memorization for Fast Learning New Knowledge without Forgetting
The ability to quickly learn new knowledge (e.g. new classes or data distributions) is a big step towards human-level intelligence. In this paper, we consider scenarios that require learning new classes or data distribut…
image-classificationImage ClassificationLanguage ModelingLanguage Modelling+1Attention-Based Structural-Plasticity
Catastrophic forgetting/interference is a critical problem for lifelong learning machines, which impedes the agents from maintaining their previously learned knowledge while learning new tasks. Neural networks, in partic…
Lifelong learningPermuted-MNISTSplit-MNISTAsynchronous Hebbian/anti-Hebbian networks
Lateral inhibition models coupled with Hebbian plasticity have been shown to learn factorised causal representations of input stimuli, for instance, oriented edges are learned from natural images. Currently, these models…
Topology-Preserving Class-Incremental Learning
A well-known issue for class-incremental learning is the catastrophic forgetting phenomenon, where the network's recognition performance on old classes degrades severely when incrementally learning new classes. To allevi…
class-incremental learningClass Incremental LearningIncremental Learning