Task-Specific Pre-Training and Cross Lingual Transfer for Code-Switched Data

2021-02-24 · Akshat Gupta, Sai Krishna Rallabandi, Alan Black

Using task-specific pre-training and leveraging cross-lingual transfer are two of the most popular ways to handle code-switched data. In this paper, we aim to compare the effects of both for the task of sentiment analysis. We work with two Dravidian Code-Switched languages - Tamil-Engish and Malayalam-English and four different BERT based models. We compare the effects of task-specific pre-training and cross-lingual transfer and find that task-specific pre-training results in superior zero-shot and supervised performance when compared to performance achieved by leveraging cross-lingual transfer from multilingual BERT models.

📄 PDF Abstract BibTeX arXiv:2102.12407

Code (0)

등록된 구현이 없습니다.

Tasks

Cross-Lingual Transfer Sentiment Analysis

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Adam 설명 없음

Multi-Head Attention 설명 없음

Residual Connection 설명 없음

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

Data-Efficient Cross-Lingual Transfer with Language-Specific Subnetworks

2022-10-31 · Rochelle Choenni, Dan Garrette, Ekaterina Shutova

Large multilingual language models typically share their parameters across all languages, which enables cross-lingual task transfer, but learning can also be hindered when training updates from different languages are in…

Cross-Lingual Transfer Meta-Learning

Exploring the BERT Cross-Lingual Transferability: a Case Study in Reading Comprehension

2020-06-17 · Proceedings of the International Conference “Dialogue 2020” 2020 6 · Konovalov V. P., Gulyaev P. A., Sorokin A. A., Kuratov Y. M. 외

Multilingual BERT has been shown to generalize well in a zero-shot cross-lingual setting. This generalization was measured on POS and NER tasks. We explore the multilingual BERT cross-language transferability on the read…

NER POS Question Answering Reading Comprehension

Parameter-efficient Zero-shot Transfer for Cross-Language Dense Retrieval with Adapters

2022-12-20 · Eugene Yang, Suraj Nair, Dawn Lawrie, James Mayfield 외

A popular approach to creating a zero-shot cross-language retrieval model is to substitute a monolingual pretrained language model in the retrieval model with a multilingual pretrained language model such as Multilingual…

Information Retrieval Language Modeling Language Modelling Retrieval

Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Models

2024-02-03 · Sara Rajaee, Christof Monz

Recent advances in training multilingual language models on large datasets seem to have shown promising results in knowledge transfer across languages and achieve high performance on downstream tasks. However, we questio…

Transfer Learning

XLM-E: Cross-lingual Language Model Pre-training via ELECTRA

2021-06-30 · ACL 2022 5 · Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma 외

In this paper, we introduce ELECTRA-style tasks to cross-lingual language model pre-training. Specifically, we present two pre-training tasks, namely multilingual replaced token detection, and translation replaced token …

Language Modeling Language Modelling Translation Zero-Shot Cross-Lingual Transfer