Improving Zero-Shot Multilingual Text Generation via Iterative Distillation
The demand for multilingual dialogue systems often requires a costly labeling process, where human translators derive utterances in low resource languages from resource rich language annotation. To this end, we explore leveraging the inductive biases for target languages learned by numerous pretrained teacher models by transferring them to student models via sequence-level knowledge distillation. By assuming no target language text, the both the teacher and student models need to learn from the target distribution in a few/zero-shot manner. On the MultiATIS++ benchmark, we explore the effectiveness of our proposed technique to derive the multilingual text for 6 languages, using only the monolingual English data and the pretrained models. We show that training on the synthetic multilingual generation outputs yields close performance to training on human annotations in both slot F1 and intent accuracy; the synthetic text also scores high in naturalness and correctness based on human evaluation.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationText GenerationSimilar Papers 제목 키워드 기반
Improving Zero-Shot Translation of Low-Resource Languages
Recent work on multilingual neural machine translation reported competitive performance with respect to bilingual models and surprisingly good performance even on (zeroshot) translation directions not observed at trainin…
Machine TranslationTranslationImproving Zero-shot Multilingual Neural Machine Translation for Low-Resource Languages
Although the multilingual Neural Machine Translation(NMT), which extends Google's multilingual NMT, has ability to perform zero-shot translation and the iterative self-learning algorithm can improve the quality of zero-s…
DiversityMachine TranslationNMTSelf-Learning+2MulZDG: Multilingual Code-Switching Framework for Zero-shot Dialogue Generation
Building dialogue generation systems in a zero-shot scenario remains a huge challenge, since the typical zero-shot approaches in dialogue generation rely heavily on large-scale pre-trained language generation models such…
Data AugmentationDialogue GenerationText GenerationLarge Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages
Recently there has been a significant surge in multimodal learning in terms of both image-to-text and text-to-image generation. However, the success is typically limited to English, leaving other languages largely behind…
Image GenerationImage to textLanguage ModelingLanguage Modelling+3The Multilingual Curse at the Retrieval Layer: Evidence from Amharic
Multilingual retrieval increasingly underpins cross-lingual question answering and retrieval-augmented generation. Strong zero-shot scores on multilingual benchmarks are often taken as evidence that current encoders tran…
Cross-Lingual Question AnsweringPassage Retrieval