CoNT: Contrastive Neural Text Generation
Recently, contrastive learning attracts increasing interests in neural text generation as a new solution to alleviate the exposure bias problem. It introduces a sequence-level training signal which is crucial to generation tasks that always rely on auto-regressive decoding. However, previous methods using contrastive learning in neural text generation usually lead to inferior performance. In this paper, we analyse the underlying reasons and propose a new Contrastive Neural Text generation framework, CoNT. CoNT addresses bottlenecks that prevent contrastive learning from being widely adopted in generation tasks from three aspects -- the construction of contrastive examples, the choice of the contrastive loss, and the strategy in decoding. We validate CoNT on five generation tasks with ten benchmarks, including machine translation, summarization, code comment generation, data-to-text generation and commonsense generation. Experimental results show that CoNT clearly outperforms the conventional training framework on all the ten benchmarks with a convincing margin. Especially, CoNT surpasses previous the most competitive contrastive learning method for text generation, by 1.50 BLEU on machine translation and 1.77 ROUGE-1 on summarization, respectively. It achieves new state-of-the-art on summarization, code comment generation (without external data) and data-to-text generation.
Code (2)
Tasks
Code Comment GenerationComment GenerationContrastive LearningData-to-Text GenerationMachine TranslationText GenerationText SummarizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generation
Decoding from the output distributions of large language models to produce high-quality text is a complex challenge in language modeling. Various approaches, such as beam search, sampling with temperature, $k-$sampling, …
DiversityLanguage ModelingLanguage ModellingText GenerationAn Empirical Study On Contrastive Search And Contrastive Decoding For Open-ended Text Generation
In the study, we empirically compare the two recently proposed decoding methods, i.e. Contrastive Search (CS) and Contrastive Decoding (CD), for open-ended text generation. The automatic evaluation results suggest that, …
DiversityText GenerationCLSEG: Contrastive Learning of Story Ending Generation
Story Ending Generation (SEG) is a challenging task in natural language generation. Recently, methods based on Pre-trained Language Models (PLM) have achieved great prosperity, which can produce fluent and coherent story…
Contrastive LearningText GenerationCoCGAN: Contrastive Learning for Adversarial Category Text Generation
The task of generating texts of different categories has attracted more and more attention in the area of natural language generation recently. Meanwhile, generative adversarial net (GAN) has demonstrated its effectivene…
Contrastive LearningText GenerationClick: Controllable Text Generation with Sequence Likelihood Contrastive Learning
It has always been an important yet challenging problem to control language models to avoid generating texts with undesirable attributes, such as toxic language and unnatural repetition. We introduce Click for controllab…
Contrastive LearningText Generation