A recurrent neural network without chaos
We introduce an exceptionally simple gated recurrent neural network (RNN) that achieves performance comparable to well-known gated architectures, such as LSTMs and GRUs, on the word-level language modeling task. We prove that our model has simple, predicable and non-chaotic dynamics. This stands in stark contrast to more standard gated architectures, whose underlying dynamical systems exhibit chaotic behavior.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
Optimal short-term memory before the edge of chaos in driven random recurrent networks
The ability of discrete-time nonlinear recurrent neural networks to store time-varying small input signals is investigated by mean-field theory. The combination of a small input strength and mean-field assumptions makes …
Harnessing and modulating chaos to sample from neural generative models
Chaos is generic in strongly-coupled recurrent networks of model neurons, and thought to be an easily accessible dynamical regime in the brain. While neural chaos is typically seen as an impediment to robust computation,…
Freezing chaos without synaptic plasticity
Chaos is ubiquitous in high-dimensional neural dynamics. A strong chaotic fluctuation may be harmful to information processing. A traditional way to mitigate this issue is to introduce Hebbian plasticity, which can stabi…
Suppression of chaos in a partially driven recurrent neural network
The dynamics of recurrent neural networks (RNNs), and particularly their response to inputs, play a critical role in information processing. In many applications of RNNs, only a specific subset of the neurons generally r…
Revisiting chaos in stimulus-driven spiking networks: signal encoding and discrimination
Highly connected recurrent neural networks often produce chaotic dynamics, meaning their precise activity is sensitive to small perturbations. What are the consequences for how such networks encode streams of temporal st…
Decoder