Conditional Language Learning with Context
Language models can learn sophisticated language understanding skills from fitting raw text. They also unselectively learn useless corpus statistics and biases, especially during finetuning on domain-specific corpora. In this paper, we propose a simple modification to causal language modeling called conditional finetuning, which performs language modeling conditioned on a context. We show that a context can "explain away" certain corpus statistics and make the model avoid learning them. In this fashion, conditional finetuning achieves selective learning from a corpus, learning knowledge useful for downstream tasks while avoiding learning useless corpus statistics like topic biases. This selective learning effect leads to less forgetting and better stability-plasticity tradeoff in domain finetuning, potentially benefitting lifelong learning with language models.
Code (1)
Tasks
Causal Language ModelingLanguage ModelingLanguage ModellingLifelong learningSimilar Papers 제목 키워드 기반
Conditional BERT Contextual Augmentation
We propose a novel data augmentation method for labeled sentences called conditional BERT contextual augmentation. Data augmentation methods are often applied to prevent overfitting and improve generalization of deep neu…
Data AugmentationLanguage ModelingLanguage ModellingText ClassificationStatic and Dynamic Vector Semantics for Lambda Calculus Models of Natural Language
Vector models of language are based on the contextual aspects of language, the distributions of words and how they co-occur in text. Truth conditional models focus on the logical aspects of language, compositional proper…
SentenceModelling Source- and Target- Language Syntactic Information as Conditional Context in Interactive Neural Machine Translation
In interactive machine translation (MT), human translators correct errors in automatic translations in collaboration with the MT systems, which is seen as an effective way to improve the productivity gain in translation.…
Machine TranslationNMTTranslationTractable Transformers for Flexible Conditional Generation
Non-autoregressive (NAR) generative models are valuable because they can handle diverse conditional generation tasks in a more principled way than their autoregressive (AR) counterparts, which are constrained by sequenti…
DecoderConditional Prompt Learning for Vision-Language Models
With the rise of powerful pre-trained vision-language models like CLIP, it becomes essential to investigate ways to adapt these models to downstream datasets. A recently proposed method named Context Optimization (CoOp) …
Domain GeneralizationPrompt EngineeringPrompt Learning