AWD-LSTM

2000년 도입 · 논문 52편에서 사용

ASGD Weight-Dropped LSTM, or AWD-LSTM, is a type of recurrent neural network that employs DropConnect for regularization, as well as NT-ASGD for optimization - non-monotonically triggered averaged SGD - which returns an average of last iterations of weights. Additional regularization techniques employed include variable length backpropagation sequences, variational dropout, embedding dropout, weight tying, independent embedding/hidden size, activation regularization and temporal activation regularization.

출처: Regularizing and Optimizing LSTM Language Models

소개 논문: Regularizing and Optimizing LSTM Language Models