What does it mean to be language-agnostic? Probing multilingual sentence encoders for typological properties
Multilingual sentence encoders have seen much success in cross-lingual model transfer for downstream NLP tasks. Yet, we know relatively little about the properties of individual languages or the general patterns of linguistic variation that they encode. We propose methods for probing sentence representations from state-of-the-art multilingual encoders (LASER, M-BERT, XLM and XLM-R) with respect to a range of typological properties pertaining to lexical, morphological and syntactic structure. In addition, we investigate how this information is distributed across all layers of the models. Our results show interesting differences in encoding linguistic variation associated with different pretraining strategies.
Code (0)
등록된 구현이 없습니다.
Tasks
SentenceXLM-RMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
When AUC 0.998 Is Not Enough: A Candidate Evaluation Protocol for Hidden-State Probes of Indirect Prompt Injection in Multimodal Computer-Use Agents
Hidden-state probing -- a linear classifier on a frozen vision-language model's internal activations -- has emerged as an attractive evaluation tool for flagging indirect prompt injection (IPI) in multimodal computer-use…
Does BERT Understand Idioms? A Probing-Based Empirical Study of BERT Encodings of Idioms
Understanding idioms is important in NLP. In this paper, we study to what extent pre-trained BERT model can encode the meaning of a potentially idiomatic expression (PIE) in a certain context. We make use of a few existi…
Paraphrase IdentificationProbing Graph Representations
Today we have a good theoretical understanding of the representational power of Graph Neural Networks (GNNs). For example, their limitations have been characterized in relation to a hierarchy of Weisfeiler-Lehman (WL) is…
DiagnosticDoes He Wink or Does He Nod? A Challenging Benchmark for Evaluating Word Understanding of Language Models
Recent progress in pretraining language models on large corpora has resulted in large performance gains on many NLP tasks. These large models acquire linguistic knowledge during pretraining, which helps to improve perfor…
Language ModelingLanguage ModellingDoes She Wink or Does She Nod? A Challenging Benchmark for Evaluating Word Understanding of Language Models
Recent progress in pretraining language models on large corpora has resulted in significant performance gains on many NLP tasks. These large models acquire linguistic knowledge during pretraining, which helps to improve …
Language ModelingLanguage Modelling