Does BERT really agree ? Fine-grained Analysis of Lexical Dependence on a Syntactic Task
Although transformer-based Neural Language Models obtain impressive results on a wide variety of tasks, their generalization abilities are not well understood. They have been shown to perform strongly on subject-verb number agreement in a wide array of settings, suggesting that they learned to capture syntactic dependencies during their training even without explicit supervision. In this paper, we examine the extent to which BERT relies on lexical content to solve the number agreement (NA) task. To do so, we disrupt the lexical patterns found in naturally occurring stimuli in a novel fine-grained analysis of BERT's behavior. Our results on nonce sentences suggest that the model generalizes well for simple structures, but fails to perform lexically-independent syntactic generalization when as little as one attractor is present.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Does BERT really agree ? Fine-grained Analysis of Lexical Dependence on a Syntactic Task
Although transformer-based Neural Language Models demonstrate impressive performance on a variety of tasks, their generalization abilities are not well understood. They have been shown to perform strongly on subject-verb…
FGR-ColBERT: Identifying Fine-Grained Relevance Tokens During Retrieval
Document retrieval identifies relevant documents but does not provide fine-grained evidence cues, such as specific relevant spans. A possible solution is to apply an LLM after retrieval; however, this introduces signific…
Does BERT agree? Evaluating knowledge of structure dependence through agreement relations
Learning representations that accurately model semantics is an important goal of natural language processing research. Many semantic phenomena depend on syntactic structure. Recent work examines the extent to which state…
SensitivityTowards Generalizeable Semantic Product Search by Text Similarity Pre-training on Search Click Logs
Recently, semantic search has been successfully applied to E-commerce product search and the learned semantic space for query and product encoding are expected to generalize well to unseen queries or products. Yet, wheth…
text similarityAblation Path Saliency
Various types of saliency methods have been proposed for explaining black-box classification. In image applications, this means highlighting the part of the image that is most relevant for the current decision. Unfortuna…