DecayNet: A Study on the Cell States of Long Short Term Memories
It is unclear whether the extensively applied long-short term memory (LSTM) is an optimised architecture for recurrent neural networks. Its complicated design makes the network hard to analyse and non-immediately clear for its utilities in real-world data. This paper studies LSTMs as systems of difference equations, and takes a theoretical mathematical approach to study consecutive transitions in network variables. Our study shows that the cell state propagation is predominantly controlled by the forget gate. Hence, we introduce DecayNets, LSTMs with monotonically decreasing forget gates, to calibrate cell state dynamics. With recurrent batch normalisation, DecayNet outperforms the previous state of the art for permuted sequential MNIST. The Decay mechanism is also beneficial for LSTM-based optimisers, and decrease optimisee neural network losses more rapidly. Edit status: Revised paper.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Temporal Performance Prediction for Deep Convolutional Long Short-Term Memory Networks
Quantifying predictive uncertainty of deep semantic segmentation networks is essential in safety-critical tasks. In applications like autonomous driving, where video data is available, convolutional long short-term memor…
Autonomous DrivingSemantic SegmentationTime SeriesSEQUENCE-LEVEL FEATURES: HOW GRU AND LSTM CELLS CAPTURE N-GRAMS
Modern recurrent neural networks (RNN) such as Gated Recurrent Units (GRU) and Long Short-term Memory (LSTM) have demonstrated impressive results on tasks involving sequential data in practice. Despite continuous efforts…
Language ModelingLanguage ModellingRolling Shutter CorrectionSentiment AnalysisOptimum Output Long Short-Term Memory Cell for High-Frequency Trading Forecasting
High-frequency trading requires fast data processing without information lags for precise stock price forecasting. This high-paced stock price forecasting is usually based on vectors that need to be treated as sequential…
Cell phenotypic transition proceeds through concerted reorganization of gene regulatory network
Phenotype transition takes place in many biological processes such as differentiation, and understanding how a cell reprograms its global gene expression profile is a problem of rate theories. A cell phenotype transition…
LARNN: Linear Attention Recurrent Neural Network
The Linear Attention Recurrent Neural Network (LARNN) is a recurrent attention module derived from the Long Short-Term Memory (LSTM) cell and ideas from the consciousness Recurrent Neural Network (RNN). Yes, it LARNNs. T…
Time SeriesTime Series Analysis