An Efficient and Effective Second-Order Training Algorithm for LSTM-based Adaptive Learning
We study adaptive (or online) nonlinear regression with Long-Short-Term-Memory (LSTM) based networks, i.e., LSTM-based adaptive learning. In this context, we introduce an efficient Extended Kalman filter (EKF) based second-order training algorithm. Our algorithm is truly online, i.e., it does not assume any underlying data generating process and future information, except that the target sequence is bounded. Through an extensive set of experiments, we demonstrate significant performance gains achieved by our algorithm with respect to the state-of-the-art methods. Here, we mainly show that our algorithm consistently provides 10 to 45\% improvement in the accuracy compared to the widely-used adaptive methods Adam, RMSprop, and DEKF, and comparable performance to EKF with a 10 to 15 times reduction in the run-time.
Code (1)
Tasks
regressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Stateless Neural Meta-Learning using Second-Order Gradients
Deep learning typically requires large data sets and much compute power for each new problem that is learned. Meta-learning can be used to learn a good prior that facilitates quick learning, thereby relaxing these requir…
image-classificationImage ClassificationMeta-LearningEfficient Second-Order TreeCRF for Neural Dependency Parsing
In the deep learning (DL) era, parsing models are extremely simplified with little hurt on performance, thanks to the remarkable capability of multi-layer BiLSTMs in context representation. As the most popular graph-base…
ARCChinese Dependency ParsingDependency ParsingReal-time detection of abnormal driving behavior based on long short-term memory network and regression residuals
Abnormal driving behavior is one of the main causes of roadway collisions. In most studies of abnormal driving behavior, the abnormal driving status is detected and analyzed using classification algorithms directly or …
Deep Differential Recurrent Neural Networks
Due to the special gating schemes of Long Short-Term Memory (LSTM), LSTMs have shown greater potential to process complex sequential information than the traditional Recurrent Neural Network (RNN). The conventional LSTM,…
Temporal SequencesDynamical System Parameter Identification using Deep Recurrent Cell Networks
In this paper, we investigate the parameter identification problem in dynamical systems through a deep learning approach. Focusing mainly on second-order, linear time-invariant dynamical systems, the topic of damping fac…