Pretraining for Conditional Generation with Pseudo Self Attention
Large pretrained language representation models have changed the way researchers approach discriminative natural language understanding tasks, leading to the dominance of approaches that finetune a pretrained model. However, such transfer learning approaches have not seen the same success for natural language generation. In this work, we explore transfer learning for conditional generation with large pretrained language models. We propose a simple modification to a pretrained unconditional transformer model to inject arbitrary conditioning into the self attention layer, an approach we call pseudo self attention. Through experiments on four long-form conditional text generation tasks, we show that this technique outperforms strong baselines and other transfer learning approaches, and produces coherent generations.
Code (0)
등록된 구현이 없습니다.
Tasks
Conditional Text GenerationNatural Language UnderstandingText GenerationTransfer LearningSimilar Papers 제목 키워드 기반
Self-Training Vision Language BERTs with a Unified Conditional Model
Natural language BERTs are trained with language corpus in a self-supervised manner. Unlike natural language BERTs, vision language BERTs need paired data to train, which restricts the scale of VL-BERT pretraining. We pr…
Nucleus-aware Self-supervised Pretraining Using Unpaired Image-to-image Translation for Histopathology Images
Self-supervised pretraining attempts to enhance model performance by obtaining effective features from unlabeled data, and has demonstrated its effectiveness in the field of histopathology images. Despite its success, fe…
DiversityImage-to-Image TranslationInstance SegmentationMultiple Instance Learning+2Encoder-Agnostic Adaptation for Conditional Language Generation
Large pretrained language models have changed the way researchers approach discriminative natural language understanding tasks, leading to the dominance of approaches that adapt a pretrained model for arbitrary downstrea…
Conditional Text GenerationLanguage ModelingLanguage ModellingNatural Language Understanding+2Conditional independence for pretext task selection in Self-supervised speech representation learning
Through solving pretext tasks, self-supervised learning (SSL) leverages unlabeled data to extract useful latent representations replacing traditional input features in the downstream task. A common pretext task consists …
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Pseudo LabelRepresentation Learning+5Neural Semi-supervised Learning for Text Classification Under Large-Scale Pretraining
The goal of semi-supervised learning is to utilize the unlabeled, in-domain dataset U to improve models trained on the labeled dataset D. Under the context of large-scale language-model (LM) pretraining, how we can make …
General ClassificationLanguage ModellingPseudo Labeltext-classification+1