Morphosyntactic probing of multilingual BERT models

2023-06-09 · Judit Acs, Endre Hamerlik, Roy Schwartz, Noah A. Smith, Andras Kornai

We introduce an extensive dataset for multilingual probing of morphological information in language models (247 tasks across 42 languages from 10 families), each consisting of a sentence with a target word and a morphological tag as the desired label, derived from the Universal Dependencies treebanks. We find that pre-trained Transformer models (mBERT and XLM-RoBERTa) learn features that attain strong performance across these tasks. We then apply two methods to locate, for each probing task, where the disambiguating information resides in the input. The first is a new perturbation method that masks various parts of context; the second is the classical method of Shapley values. The most intriguing finding that emerges is a strong tendency for the preceding context to hold more information relevant to the prediction than the following context.

📄 PDF Abstract BibTeX arXiv:2306.06205

Code (1)

juditacs/morphology-probes 공식 구현

Tasks

SentenceTAG

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Residual Connection 설명 없음

A multilabel approach to morphosyntactic probing

2021-04-17 · Findings (EMNLP) 2021 11 · Naomi Tachikawa Shapiro, Amandalynne Paullada, Shane Steinert-Threlkeld

We introduce a multilabel probing task to assess the morphosyntactic representations of word embeddings from multilingual language models. We demonstrate this task with multilingual BERT (Devlin et al., 2018), training p…

Word Embeddings

Universal and Independent: Multilingual Probing Framework for Exhaustive Model Interpretation and Evaluation

2022-10-24 · Oleg Serikov, Vitaly Protasov, Ekaterina Voloshina, Viktoria Knyazkova 외

Linguistic analysis of language models is one of the ways to explain and describe their reasoning, weaknesses, and limitations. In the probing part of the model interpretability research, studies concern individual langu…

Probing Language Models

Morph Call: Probing Morphosyntactic Content of Multilingual Transformers

2021-04-26 · NAACL (SIGTYP) 2021 6 · Vladislav Mikhailov, Oleg Serikov, Ekaterina Artemova

The outstanding performance of transformer-based language models on a great variety of NLP and NLU tasks has stimulated interest in exploring their inner workings. Recent research has focused primarily on higher-level an…

Common Sense Reasoning MORPH POS POS Tagging+2

A Joint Matrix Factorization Analysis of Multilingual Representations

2023-10-24 · Zheng Zhao, Yftah Ziser, Bonnie Webber, Shay B. Cohen

We present an analysis tool based on joint matrix factorization for comparing latent representations of multilingual and monolingual models. An alternative to probing, this tool allows us to analyze multiple sets of repr…

Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERT

2021-01-26 · EACL 2021 2 · Isabel Papadimitriou, Ethan A. Chi, Richard Futrell, Kyle Mahowald

We investigate how Multilingual BERT (mBERT) encodes grammar by examining how the high-order grammatical feature of morphosyntactic alignment (how different languages define what counts as a "subject") is manifested acro…

Sentence