Persistence pays off: Paying Attention to What the LSTM Gating Mechanism Persists
Language Models (LMs) are important components in several Natural Language Processing systems. Recurrent Neural Network LMs composed of LSTM units, especially those augmented with an external memory, have achieved state-of-the-art results. However, these models still struggle to process long sequences which are more likely to contain long-distance dependencies because of information fading and a bias towards more recent information. In this paper we demonstrate an effective mechanism for retrieving information in a memory augmented LSTM LM based on attending to information in memory in proportion to the number of timesteps the LSTM gating mechanism persisted the information.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Persistence pays off: Paying Attention to What the LSTM Gating Mechanism Persists
Recurrent Neural Network Language Models composed of LSTM units, especially those augmented with an external memory, have achieved state-of-the-art results in Language Modeling. However, these models still struggle to pr…
Language ModelingLanguage ModellingPaying and Persuading
I study the joint design of information and transfers when an informed Sender can motivate Receiver by both paying and (Bayesian) persuading. I introduce a generalized concavification method to characterize Sender's valu…
InformativenessVerifying Attention Robustness of Deep Neural Networks against Semantic Perturbations
It is known that deep neural networks (DNNs) classify an input image by paying particular attention to certain specific pixels; a graphical representation of the magnitude of attention to each pixel is called a saliency-…
validPaying Attention to Deflections: Mining Pragmatic Nuances for Whataboutism Detection in Online Discourse
Whataboutism, a potent tool for disrupting narratives and sowing distrust, remains under-explored in quantitative NLP research. Moreover, past work has not distinguished its use as a strategy for misinformation and propa…
MisinformationOptimal valuation of American callable credit default swaps under drawdown of L\'evy insurance risk process
This paper discusses the valuation of credit default swaps, where default is announced when the reference asset price has gone below certain level from the last record maximum, also known as the high-water mark or drawdo…