LSTMs Exploit Linguistic Attributes of Data
While recurrent neural networks have found success in a variety of natural language processing applications, they are general models of sequential data. We investigate how the properties of natural language data affect an LSTM's ability to learn a nonlinguistic task: recalling elements from its input. We find that models trained on natural language data are able to recall tokens from much longer sequences than models trained on non-language sequential data. Furthermore, we show that the LSTM learns to solve the memorization task by explicitly using a subset of its neurons to count timesteps in the input. We hypothesize that the patterns and structure in natural language data enable LSTMs to learn by providing approximate ways of reducing loss, but understanding the effect of different training data on the learnability of LSTMs remains an open question.
Code (0)
등록된 구현이 없습니다.
Tasks
MemorizationOpen-Ended Question AnsweringMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Do LSTMs See Gender? Probing the Ability of LSTMs to Learn Abstract Syntactic Rules
LSTMs trained on next-word prediction can accurately perform linguistic tasks that require tracking long-distance syntactic dependencies. Notably, model accuracy approaches human performance on number agreement tasks (Gu…
ArticlesVideo Captioning with Multi-Faceted Attention
Recently, video captioning has been attracting an increasing amount of interest, due to its potential for improving accessibility and information retrieval. While existing methods rely on different kinds of visual featur…
Information RetrievalRetrievalSentenceVideo CaptioningCLiMP: A Benchmark for Chinese Language Model Evaluation
Linguistically informed analyses of language models (LMs) contribute to the understanding and improvement of these models. Here, we introduce the corpus of Chinese linguistic minimal pairs (CLiMP), which can be used to i…
Language Model EvaluationLanguage ModelingLanguage ModellingmodelFrom Visual Attributes to Adjectives through Decompositional Distributional Semantics
As automated image analysis progresses, there is increasing interest in richer linguistic annotation of pictures, with attributes of objects (e.g., furry, brown...) attracting most attention. By building on the recent "z…
AttributeObjectObject RecognitionRetrieval+2Accurate 3D Body Shape Regression using Metric and Semantic Attributes
While methods that regress 3D human meshes from images have progressed rapidly, the estimated body shapes often do not capture the true human shape. This is problematic since, for many applications, accurate body shape i…
3D Human Reconstruction3D Human Shape Estimation