Large-Scale Transfer Learning for Natural Language Generation
Large-scale pretrained language models define state of the art in natural language processing, achieving outstanding performance on a variety of tasks. We study how these architectures can be applied and adapted for natural language generation, comparing a number of architectural and training schemes. We focus in particular on open-domain dialog as a typical high entropy generation task, presenting and comparing different architectures for adapting pretrained models with state of the art results.
Code (1)
Tasks
Open-Domain DialogText GenerationTransfer LearningSimilar Papers 제목 키워드 기반
Pretraining for Conditional Generation with Pseudo Self Attention
Large pretrained language representation models have changed the way researchers approach discriminative natural language understanding tasks, leading to the dominance of approaches that finetune a pretrained model. Howe…
Conditional Text GenerationNatural Language UnderstandingText GenerationTransfer LearningEffect of Pre-Training Scale on Intra- and Inter-Domain Full and Few-Shot Transfer Learning for Natural and Medical X-Ray Chest Images
Increasing model, data and compute budget scale in the pre-training has been shown to strongly improve model generalization and transfer learning in vast line of work done in language modeling and natural image recogniti…
Few-Shot LearningImage ClassificationLanguage ModelingLanguage Modelling+1Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?
Recent advancements in Large Language Models (LLMs) have drawn increasing attention since the learned embeddings pretrained on large-scale datasets have shown powerful ability in various downstream applications. However,…
Electrocardiography (ECG)zero-shot-classificationZero-Shot LearningPyMT5: multi-mode translation of natural language and Python code with transformers
Simultaneously modeling source code and natural language has many exciting applications in automated software development and understanding. Pursuant to achieving such technology, we introduce PyMT5, the Python method te…
TranslationCross-lingual Transfer Learning with Data Selection for Large-Scale Spoken Language Understanding
A typical cross-lingual transfer learning approach boosting model performance on a language is to pre-train the model on all available supervised data from another language. However, in large-scale systems this leads to …
Cross-Lingual TransferLanguage ModelingLanguage ModellingSpoken Language Understanding+1