Learning Sparse, Distributed Representations using the Hebbian Principle
The "fire together, wire together" Hebbian model is a central principle for learning in neuroscience, but surprisingly, it has found limited applicability in modern machine learning. In this paper, we take a first step towards bridging this gap, by developing flavors of competitive Hebbian learning which produce sparse, distributed neural codes using online adaptation with minimal tuning. We propose an unsupervised algorithm, termed Adaptive Hebbian Learning (AHL). We illustrate the distributed nature of the learned representations via output entropy computations for synthetic data, and demonstrate superior performance, compared to standard alternatives such as autoencoders, in training a deep convolutional net on standard image datasets.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Can a Hebbian-like learning rule be avoiding the curse of dimensionality in sparse distributed data?
It is generally assumed that the brain uses something akin to sparse distributed representations. These representations, however, are high-dimensional and consequently they affect classification performance of traditiona…
Learning Successor Features with Distributed Hebbian Temporal Memory
This paper presents a novel approach to address the challenge of online temporal memory learning for decision-making under uncertainty in non-stationary, partially observable environments. The proposed algorithm, Distrib…
Decision MakingDecision Making Under UncertaintyRepresentation LearningNonlinear Hebbian learning as a unifying principle in receptive field formation
The development of sensory receptive fields has been modeled in the past by a variety of models including normative models such as sparse coding or independent component analysis and bottom-up models such as spike-timing…
Hebbian Continual Representation Learning
Continual Learning aims to bring machine learning into a more realistic scenario, where tasks are learned sequentially and the i.i.d. assumption is not preserved. Although this setting is natural for biological systems, …
BIG-bench Machine Learningclass-incremental learningClass Incremental LearningContinual Learning+2Brain-like combination of feedforward and recurrent network components achieves prototype extraction and robust pattern recognition
Associative memory has been a prominent candidate for the computation performed by the massively recurrent neocortical networks. Attractor networks implementing associative memory have offered mechanistic explanation for…