Alleviating Exposure Bias via Multi-level Contrastive Learning and Deviation Simulation in Abstractive Summarization
Most Transformer based abstractive summarization systems have a severe mismatch between training and inference, i.e., exposure bias. From diverse perspectives, we introduce a simple multi-level contrastive learning framework for abstractive summarization (SimMCS) and a tailored sparse decoder self-attention pattern (SDSA) to bridge the gap between training and inference to improve model performance. Compared with previous contrastive objectives focusing only on the relative order of probability mass assigned to non-gold summaries, SimMCS additionally takes their absolute positions into account, which guarantees that the relatively high-quality (positive) summaries among them could be properly assigned high probability mass, and further enhances the capability of discriminating summary quality beyond exploiting potential artifacts of specific metrics. SDSA simulates the possible inference scenarios of deviation in the training phase to get closer to the ideal paradigm. Our approaches outperform the previous state-of-the-art results on two summarization datasets while just adding fairly low overhead. Further empirical analysis shows our model preserves the advantages of prior contrastive methods and possesses strong few-shot learning ability.
Code (1)
Tasks
Abstractive Text SummarizationContrastive LearningDecoderFew-Shot LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Alleviating Exposure Bias via Contrastive Learning for Abstractive Text Summarization
Encoder-decoder models have achieved remarkable success in abstractive text summarization, which aims to compress one or more documents into a shorter version without the loss of the essential content. Unfortunately, the…
Abstractive Text SummarizationContrastive LearningDecoderText SummarizationContrastive Learning for Debiased Candidate Generation in Large-Scale Recommender Systems
Deep candidate generation (DCG) that narrows down the collection of relevant items from billions to hundreds via representation learning has become prevalent in industrial recommender systems. Standard approaches approxi…
Contrastive LearningFairnessLanguage ModelingLanguage Modelling+2Contrastive Reinforced Policy Optimization via Privileged Self-Distillation
Recent advances in post-training Large Language Models (LLMs) increasingly rely on Reinforcement Learning with Verifiable Rewards (RLVR) or On-Policy Self-Distillation (OPSD). While OPSD provides dense, logit-level super…
Reinforcement LearningContrastive LearningOrder-free Learning Alleviating Exposure Bias in Multi-label Classification
Multi-label classification (MLC) assigns multiple labels to each sample. Prior studies show that MLC can be transformed to a sequence prediction problem with a recurrent neural network (RNN) decoder to model the label de…
DecoderGeneral ClassificationMulti-Label ClassificationMUlTI-LABEL-ClASSIFICATIONQuantifying the Potential to Escape Filter Bubbles: A Behavior-Aware Measure via Contrastive Simulation
Nowadays, recommendation systems have become crucial to online platforms, shaping user exposure by accurate preference modeling. However, such an exposure strategy can also reinforce users' existing preferences, leading …
Recommendation Systems