A Joint Matrix Factorization Analysis of Multilingual Representations
We present an analysis tool based on joint matrix factorization for comparing latent representations of multilingual and monolingual models. An alternative to probing, this tool allows us to analyze multiple sets of representations in a joint manner. Using this tool, we study to what extent and how morphosyntactic features are reflected in the representations learned by multilingual pre-trained models. We conduct a large-scale empirical study of over 33 languages and 17 morphosyntactic categories. Our findings demonstrate variations in the encoding of morphosyntactic information across upper and lower layers, with category-specific differences influenced by language properties. Hierarchical clustering of the factorization outputs yields a tree structure that is related to phylogenetic trees manually crafted by linguists. Moreover, we find the factorization outputs exhibit strong associations with performance observed across different cross-lingual tasks. We release our code to facilitate future research.
Code (1)
Similar Papers 제목 키워드 기반
Learning From Hidden Traits: Joint Factor Analysis and Latent Clustering
Dimensionality reduction techniques play an essential role in data analytics, signal processing and machine learning. Dimensionality reduction is usually performed in a preprocessing stage that is separate from subsequen…
ClusteringDimensionality ReductionApplications of Online Nonnegative Matrix Factorization to Image and Time-Series Data
Online nonnegative matrix factorization (ONMF) is a matrix factorization technique in the online setting where data are acquired in a streaming fashion and the matrix factors are updated each time. This enables factor an…
Dictionary LearningTime SeriesTime Series AnalysisRethinking Non-Negative Matrix Factorization with Implicit Neural Representations
Non-negative Matrix Factorization (NMF) is a powerful technique for analyzing regularly-sampled data, i.e., data that can be stored in a matrix. For audio, this has led to numerous applications using time-frequency (TF) …
Boosting the Rating Prediction with Click Data and Textual Contents
Matrix factorization (MF) is one of the most efficient methods for rating predictions. MF learns user and item representations by factorizing the user-item rating matrix. Further, textual contents are integrated to conve…
A Simple and Effective Method To Eliminate the Self Language Bias in Multilingual Representations
Language agnostic and semantic-language information isolation is an emerging research direction for multilingual representations models. We explore this problem from a novel angle of geometric algebra and semantic space.…
Cross-Lingual TransferRetrieval