Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing
This paper investigates the problem of learning cross-lingual representations in a contextual space. We propose Cross-Lingual BERT Transformation (CLBT), a simple and efficient approach to generate cross-lingual contextualized word embeddings based on publicly available pre-trained BERT models (Devlin et al., 2018). In this approach, a linear transformation is learned from contextual word alignments to align the contextualized embeddings independently trained in different languages. We demonstrate the effectiveness of this approach on zero-shot cross-lingual transfer parsing. Experiments show that our embeddings substantially outperform the previous state-of-the-art that uses static embeddings. We further compare our approach with XLM (Lample and Conneau, 2019), a recently proposed cross-lingual language model trained with massive parallel data, and achieve highly competitive results.
Code (1)
Tasks
Cross-Lingual TransferDependency ParsingLanguage ModelingLanguage ModellingWord EmbeddingsZero-Shot Cross-Lingual TransferMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Linear Transformations for Cross-lingual Sentiment Analysis
This paper deals with cross-lingual sentiment analysis in Czech, English and French languages. We perform zero-shot cross-lingual classification using five linear transformations combined with LSTM and CNN based classifi…
ClassificationSentiment AnalysisFeature Aggregation in Zero-Shot Cross-Lingual Transfer Using Multilingual BERT
Multilingual BERT (mBERT), a language model pre-trained on large multilingual corpora, has impressive zero-shot cross-lingual transfer capabilities and performs surprisingly well on zero-shot POS tagging and Named Entity…
Cross-Lingual TransferLanguage ModelingLanguage Modellingnamed-entity-recognition+6Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT
Pretrained contextual representation models (Peters et al., 2018; Devlin et al., 2018) have pushed forward the state-of-the-art on many NLP tasks. A new release of BERT (Devlin, 2018) includes a model simultaneously pret…
Cross-Lingual NERCross-Lingual TransferDependency ParsingDocument Classification+5Examining Cross-lingual Contextual Embeddings with Orthogonal Structural Probes
State-of-the-art contextual embeddings are obtained from large language models available only for a few languages. For others, we need to learn representations using a multilingual model. There is an ongoing debate on wh…
An Empirical Study of Pre-trained Transformers for Arabic Information Extraction
Multilingual pre-trained Transformers, such as mBERT (Devlin et al., 2019) and XLM-RoBERTa (Conneau et al., 2020a), have been shown to enable the effective cross-lingual zero-shot transfer. However, their performance on …
Cross-Lingual TransferLanguage Modellingnamed-entity-recognitionNamed Entity Recognition+8