How Distributed are Distributed Representations? An Observation on the Locality of Syntactic Information in Verb Agreement Tasks
This work addresses the question of the localization of syntactic information encoded in the transformers representations. We tackle this question from two perspectives, considering the object-past participle agreement in French, by identifying, first, in which part of the sentence and, second, in which part of the representation the syntactic information is encoded. The results of our experiments, using probing, causal analysis and feature selection method, show that syntactic information is encoded locally in a way consistent with the French grammar.
Code (0)
등록된 구현이 없습니다.
Tasks
feature selectionSentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
How Distributed are Distributed Representations? An Observation on the Locality of Syntactic Information in Verb Agreement Tasks
This work addresses the question of the localization of the syntactic information encoding in the Transformers representations. We tackle this question from two perspectives, considering the object-past participle agreem…
feature selectionSentenceLes représentations distribuées sont-elles vraiment distribuées ? Observations sur la localisation de l’information syntaxique dans les tâches d’accord du verbe en français (How Distributed are Distributed Representations ? An Observation on the Locality of Syntactic)
Ce travail aborde la question de la localisation de l’information syntaxique qui est encodée dans les représentations de transformers. En considérant la tâche d’accord objet-participe passé en français, les résultats de …
Leveraging distributed representations and lexico-syntactic fixedness for token-level prediction of the idiomaticity of English verb-noun combinations
Verb-noun combinations (VNCs) - e.g., blow the whistle, hit the roof, and see stars - are a common type of English idiom that are ambiguous with literal usages. In this paper we propose and evaluate models for classifyin…
Machine TranslationSentence EmbeddingsWord EmbeddingsSyntactic Dependencies and Distributed Word Representations for Analogy Detection and Mining
AILA--First Experiments with Localist Language Models
This paper presents the first empirical demonstration of controllable locality in transformer language models, a novel architectural framework that enables continuous control over the degree of representation localizatio…
Continuous Control