Unsupervised Pretraining for Sequence to Sequence Learning
This work presents a general unsupervised learning method to improve the accuracy of sequence to sequence (seq2seq) models. In our method, the weights of the encoder and decoder of a seq2seq model are initialized with the pretrained weights of two language models and then fine-tuned with labeled data. We apply this method to challenging benchmarks in machine translation and abstractive summarization and find that it significantly improves the subsequent supervised models. Our main result is that pretraining improves the generalization of seq2seq models. We achieve state-of-the-art results on the WMT English→German task, surpassing a range of methods using both phrase-based machine translation and neural machine translation. Our method achieves a significant improvement of 1.3 BLEU from th previous best models on both WMT{'}14 and WMT{'}15 English→German. We also conduct human evaluations on abstractive summarization and find that our method outperforms a purely supervised learning baseline in a statistically significant manner.
Code (0)
등록된 구현이 없습니다.
Tasks
Abstractive Text SummarizationDecoderLanguage ModelingLanguage ModellingMachine TranslationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling
Contextualized word embeddings such as ELMo and BERT provide a foundation for strong performance across a wide range of natural language processing tasks by pretraining on large corpora of unlabeled text. However, the ap…
Domain AdaptationLanguage ModelingLanguage ModellingMasked Language Modeling+3Unsupervised Pretraining for Sequence to Sequence Learning
This work presents a general unsupervised learning method to improve the accuracy of sequence to sequence (seq2seq) models. In our method, the weights of the encoder and decoder of a seq2seq model are initialized with th…
Abstractive Text SummarizationDecoderMachine TranslationTranslationSemi-supervised Sequence Learning
We present two approaches that use unlabeled data to improve sequence learning with recurrent networks. The first approach is to predict what comes next in a sequence, which is a conventional language model in natural la…
Language ModelingLanguage ModellingText ClassificationContinuous Sign Language Recognition with Adapted Conformer via Unsupervised Pretraining
Conventional Deep Learning frameworks for continuous sign language recognition (CSLR) are comprised of a single or multi-modal feature extractor, a sequence-learning module, and a decoder for outputting the glosses. The …
Sign Language Recognitionspeech-recognitionSpeech RecognitionExploring Unsupervised Pretraining Objectives for Machine Translation
Unsupervised cross-lingual pretraining has achieved strong results in neural machine translation (NMT), by drastically reducing the need for large parallel data. Most approaches adapt masked-language modeling (MLM) to se…
DecoderLanguage ModelingLanguage ModellingMachine Translation+3