Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State
We deploy the methods of controlled psycholinguistic experimentation to shed light on the extent to which the behavior of neural network language models reflects incremental representations of syntactic state. To do so, we examine model behavior on artificial sentences containing a variety of syntactically complex structures. We test four models: two publicly available LSTM sequence models of English (Jozefowicz et al., 2016; Gulordava et al., 2018) trained on large datasets; an RNNG (Dyer et al., 2016) trained on a small, parsed dataset; and an LSTM trained on the same small corpus as the RNNG. We find evidence that the LSTMs trained on large datasets represent syntactic state over large spans of text in a way that is comparable to the RNNG, while the LSTM trained on the small dataset does not or does so only weakly.
Code (2)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency
Recurrent neural networks (RNNs) are the state of the art in sequence modeling for natural language. However, it remains poorly understood what grammatical characteristics of natural language they implicitly learn and re…
Language ModelingLanguage ModellingUsing Priming to Uncover the Organization of Syntactic Representations in Neural Language Models
Neural language models (LMs) perform well on tasks that require sensitivity to syntactic structure. Drawing on the syntactic priming paradigm from psycholinguistics, we propose a novel technique to analyze the representa…
SensitivitySentenceLarge Language Models Are Partially Primed in Pronoun Interpretation
While a large body of literature suggests that large language models (LLMs) acquire rich linguistic representations, little is known about whether they adapt to linguistic biases in a human-like way. The present study pr…
In-Context LearningLarge Language Models as Neurolinguistic Subjects: Identifying Internal Representations for Form and Meaning
This study investigates the linguistic understanding of Large Language Models (LLMs) regarding signifier (form) and signified (meaning) by distinguishing two LLM evaluation paradigms: psycholinguistic and neurolinguistic…
DiagnosticFormDiscourse structure interacts with reference but not syntax in neural language models
Language models (LMs) trained on large quantities of text have been claimed to acquire abstract linguistic representations. Our work tests the robustness of these abstractions by focusing on the ability of LMs to learn i…
coreference-resolutionCoreference ResolutionLanguage ModelingLanguage Modelling