Does BERT agree? Evaluating knowledge of structure dependence through agreement relations
Learning representations that accurately model semantics is an important goal of natural language processing research. Many semantic phenomena depend on syntactic structure. Recent work examines the extent to which state-of-the-art models for pre-training representations, such as BERT, capture such structure-dependent phenomena, but is largely restricted to one phenomenon in English: number agreement between subjects and verbs. We evaluate BERT's sensitivity to four types of structure-dependent agreement relations in a new semi-automatically curated dataset across 26 languages. We show that both the single-language and multilingual BERT models capture syntax-sensitive agreement patterns well in general, but we also highlight the specific linguistic contexts in which their performance degrades.
Code (1)
Tasks
SensitivityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Open Sesame: Getting Inside BERT's Linguistic Knowledge
How and to what extent does BERT encode syntactically-sensitive hierarchical information or positionally-sensitive linear information? Recent work has shown that contextual representations like BERT perform well on tasks…
DiagnosticSensitivityDoes BERT really agree ? Fine-grained Analysis of Lexical Dependence on a Syntactic Task
Although transformer-based Neural Language Models obtain impressive results on a wide variety of tasks, their generalization abilities are not well understood. They have been shown to perform strongly on subject-verb num…
Does BERT really agree ? Fine-grained Analysis of Lexical Dependence on a Syntactic Task
Although transformer-based Neural Language Models demonstrate impressive performance on a variety of tasks, their generalization abilities are not well understood. They have been shown to perform strongly on subject-verb…
What Does BERT Learn about the Structure of Language?
BERT is a recent language representation model that has surprisingly performed well in diverse language understanding benchmarks. This result indicates the possibility that BERT networks capture structural information ab…
Subject Verb Agreement Error Patterns in Meaningless Sentences: Humans vs. BERT
Both humans and neural language models are able to perform subject-verb number agreement (SVA). In principle, semantics shouldn't interfere with this task, which only requires syntactic knowledge. In this work we test wh…