From Monolingual to Multilingual FAQ Assistant using Multilingual Co-training
Recent research on cross-lingual transfer show state-of-the-art results on benchmark datasets using pre-trained language representation models (PLRM) like BERT. These results are achieved with the traditional training approaches, such as Zero-shot with no data, Translate-train or Translate-test with machine translated data. In this work, we propose an approach of {``}Multilingual Co-training{''} (MCT) where we augment the expert annotated dataset in the source language (English) with the corresponding machine translations in the target languages (e.g. Arabic, Spanish) and fine-tune the PLRM jointly. We observe that the proposed approach provides consistent gains in the performance of BERT for multiple benchmark datasets (e.g. 1.0{\%} gain on MLDocs, and 1.2{\%} gain on XNLI over translate-train with BERT), while requiring a single model for multiple languages. We further consider a FAQ dataset where the available English test dataset is translated by experts into Arabic and Spanish. On such a dataset, we observe an average gain of 4.9{\%} over all other cross-lingual transfer protocols with BERT. We further observe that domain-specific joint pre-training of the PLRM using HR policy documents in English along with the machine translations in the target languages, followed by the joint finetuning, provides a further improvement of 2.8{\%} in average accuracy.
Code (0)
등록된 구현이 없습니다.
Tasks
Cross-Lingual TransferMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca
Foundational large language models (LLMs) can be instruction-tuned to perform open-domain question answering, facilitating applications like chat assistants. While such efforts are often carried out in a single language,…
Instruction FollowingLarge Language ModelMultilingual NLPOpen-Domain Question Answering+2Evaluating Cross-Lingual Transfer Learning Approaches in Multilingual Conversational Agent Models
With the recent explosion in popularity of voice assistant devices, there is a growing interest in making them available to user populations in additional countries and languages. However, to provide the highest accuracy…
Cross-Lingual TransferNatural Language UnderstandingTransfer LearningA System for Multilingual Dependency Parsing based on Bidirectional LSTM Feature Representations
In this paper, we present our multilingual dependency parser developed for the CoNLL 2017 UD Shared Task dealing with {``}Multilingual Parsing from Raw Text to Universal Dependencies{''}. Our parser extends the monolingu…
Dependency ParsingMultilingual Word EmbeddingsWord EmbeddingsHow Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models
In this work, we provide a systematic and comprehensive empirical comparison of pretrained multilingual language models versus their monolingual counterparts with regard to their monolingual task performance. We study a …
Pretrained Multilingual Language ModelsA Checkpoint on Multilingual Misogyny Identification
We address the problem of identifying misogyny in tweets in mono and multilingual settings in three languages: English, Italian, and Spanish. We explore model variations considering single and multiple languages both in …
Transfer Learningzero-shot-classificationZero-Shot Learning