Associative Long Short-Term Memory

2016-02-09 · Ivo Danihelka, Greg Wayne, Benigno Uria, Nal Kalchbrenner, Alex Graves

We investigate a new method to augment recurrent neural networks with extra memory without increasing the number of network parameters. The system has an associative memory based on complex-valued vectors and is closely related to Holographic Reduced Representations and Long Short-Term Memory networks. Holographic Reduced Representations have limited capacity: as they store more information, each retrieval becomes noisier due to interference. Our system in contrast creates redundant copies of stored information, which enables retrieval with reduced noise. Experiments demonstrate faster learning on multiple memorization tasks.

📄 PDF Abstract BibTeX arXiv:1602.03032

Code (3)

henrysteinitz/holographic-reduced-representations pytorch

henrysteinitz/neural-memory pytorch

mohammadpz/Associative_LSTM

Tasks

Memorization Retrieval

Methods 이 논문이 사용한 방법론

Sigmoid Activation 설명 없음

Tanh Activation 설명 없음

Holographic Reduced Representation Holographic Reduced Representations are a simple mechanism to represent an associative array of key-value pairs in a fixed-size vector. Each individual key-value pair is the…

LSTM An LSTM is a type of recurrent neural network that addresses the vanishing gradient problem in vanilla…

Adam 설명 없음

Associative LSTM An Associative LSTM combines an LSTM with ideas from Holographic Reduced Representations (HRRs) to enable key-value storage of data.…

Fast Weight Long Short-Term Memory

2018-04-18 · T. Anderson Keller, Sharath Nittur Sridhar, Xin Wang

Associative memory using fast weights is a short-term memory mechanism that substantially improves the memory capacity and time scale of recurrent neural networks (RNNs). As recent studies introduced fast weights only to…

Retrieval

Learning to update Auto-associative Memory in Recurrent Neural Networks for Improving Sequence Memorization

2017-09-19 · Wei Zhang, Bo-Wen Zhou

Learning to remember long sequences remains a challenging task for recurrent neural networks. Register memory and attention mechanisms were both proposed to resolve the issue with either high computational cost to retain…

Memorization Representation Learning

'Memory States' from Almost Nothing: Representing and Computing in a Non-associative Algebra

2025-05-13 · Stefan Reimann

This note presents a non-associative algebraic framework for the representation and computation of information items in high-dimensional space. This framework is consistent with the principles of spatial computing and wi…

Position

GATED FAST WEIGHTS FOR ASSOCIATIVE RETRIEVAL

2018-01-01 · ICLR 2018 1 · Imanol Schlag, Jürgen Schmidhuber

We improve previous end-to-end differentiable neural networks (NNs) with fast weight memories. A gate mechanism updates fast weights at every time step of a sequence through two separate outer-product-based matrices gene…