What do RNN Language Models Learn about Filler--Gap Dependencies?
RNN language models have achieved state-of-the-art perplexity results and have proven useful in a suite of NLP tasks, but it is as yet unclear what syntactic generalizations they learn. Here we investigate whether state-of-the-art RNN language models represent long-distance \textbf{filler{--}gap dependencies} and constraints on them. Examining RNN behavior on experimentally controlled sentences designed to expose filler{--}gap dependencies, we show that RNNs can represent the relationship in multiple syntactic positions and over large spans of text. Furthermore, we show that RNNs learn a subset of the known restrictions on filler{--}gap dependencies, known as \textbf{island constraints}: RNNs show evidence for wh-islands, adjunct islands, and complex NP islands. These studies demonstrates that state-of-the-art RNN models are able to learn and generalize about empty syntactic positions.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingMachine TranslationSimilar Papers 제목 키워드 기반
What do RNN Language Models Learn about Filler-Gap Dependencies?
RNN language models have achieved state-of-the-art perplexity results and have proven useful in a suite of NLP tasks, but it is as yet unclear what syntactic generalizations they learn. Here we investigate whether state-…
What Don’t RNN Language Models Learn About Filler-Gap Dependencies?
Filler-gaps that neural networks fail to generalize
It can be difficult to separate abstract linguistic knowledge in recurrent neural networks (RNNs) from surface heuristics. In this work, we probe for highly abstract syntactic constraints that have been claimed to govern…
What Syntactic Structures block Dependencies in RNN Language Models?
Recurrent Neural Networks (RNNs) trained on a language modeling task have been shown to acquire a number of non-local grammatical dependencies with some success. Here, we provide new evidence that RNN language models are…
Language ModelingLanguage ModellingGeneralizations across filler-gap dependencies in neural language models
Humans develop their grammars by making structural generalizations from finite input. We ask how filler-gap dependencies, which share a structural generalization despite diverse surface forms, might arise from the input.…
Language AcquisitionLanguage ModelingLanguage Modelling