Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages
The impressive achievements of transformers force NLP researchers to delve into how these models represent the underlying structure of natural language. In this paper, we propose a novel standpoint to investigate the above issue: using typological similarities among languages to observe how their respective monolingual models encode structural information. We aim to layer-wise compare transformers for typologically similar languages to observe whether these similarities emerge for particular layers. For this investigation, we propose to use Centered Kernel Alignment to measure similarity among weight matrices. We found that syntactic typological similarity is consistent with the similarity between the weights in the middle layers, which are the pretrained BERT layers to which syntax encoding is generally attributed. Moreover, we observe that a domain adaptation on semantically equivalent texts enhances this similarity among weight matrices.
Code (0)
등록된 구현이 없습니다.
Tasks
Domain AdaptationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
The Past, Present, and Future of Typological Databases in NLP
Typological information has the potential to be beneficial in the development of NLP models, particularly for low-resource languages. Unfortunately, current large-scale typological databases, notably WALS and Grambank, a…
Language ModelingLanguage ModellingAre All Good Word Vector Spaces Isomorphic?
Existing algorithms for aligning cross-lingual word vector spaces assume that vector spaces are approximately isomorphic. As a result, they perform poorly or fail completely on non-isomorphic spaces. Such non-isomorphism…
AllInvestigating Language Relationships in Multilingual Sentence Encoders Through the Lens of Linguistic Typology
Multilingual sentence encoders have seen much success in cross-lingual model transfer for downstream NLP tasks. The success of this transfer is, however, dependent on the model’s ability to encode the patterns of cross-l…
SentenceXLM-RToward Universal Dependencies for Shipibo-Konibo
We present an initial version of the Universal Dependencies (UD) treebank for Shipibo-Konibo, the first South American, Amazonian, Panoan and Peruvian language with a resource built under UD. We describe the linguistic a…
Dependency ParsingMachine TranslationAssessing the Impact of Typological Features on Multilingual Machine Translation in the Age of Large Language Models
Despite major advances in multilingual modeling, large quality disparities persist across languages. Besides the obvious impact of uneven training resources, typological properties have also been proposed to determine th…
Machine Translation