Diverse Preference Optimization
Post-training of language models, either through reinforcement learning, preference optimization or supervised finetuning, tends to sharpen the output probability distribution and reduce the diversity of generated responses. This is particularly a problem for creative generative tasks where varied responses are desired. In this work we introduce Diverse Preference Optimization (DivPO), an optimization method which learns to generate much more diverse responses than standard pipelines, while maintaining the quality of the generations. In DivPO, preference pairs are selected by first considering a pool of responses, and a measure of diversity among them, and selecting chosen examples as being more rare but high quality, while rejected examples are more common, but low quality. DivPO results in generating 45.6% more diverse persona attributes, and an 74.6% increase in story diversity, while maintaining similar win rates as standard baselines.
Code (1)
Tasks
DiversitySimilar Papers 제목 키워드 기반
BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization
While learning to align Large Language Models (LLMs) with human preferences has shown remarkable success, aligning these models to meet the diverse user preferences presents further challenges in preserving previous know…
Continual LearningGeneral KnowledgeKnowledge DistillationPersonalized and Emotional Conversation+1Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts
In the field of large language models (LLMs), aligning models with the diverse preferences of users is a critical challenge. Direct Preference Optimization (DPO) has played a key role in this area. It works by using pair…
FairPO: Robust Preference Optimization for Fair Multi-Label Learning
We propose FairPO, a novel framework designed to promote fairness in multi-label classification by directly optimizing preference signals with a group robustness perspective. In our framework, the set of labels is partit…
FairnessMulti-Label ClassificationMUlTI-LABEL-ClASSIFICATIONMulti-Label LearningAnnotation-Efficient Preference Optimization for Language Model Alignment
Preference optimization is a standard approach to fine-tuning large language models to align with human preferences. The quality, diversity, and quantity of the preference dataset are critical to the effectiveness of pre…
DiversityLanguage ModelingLanguage ModellingmodelDirect Preference Optimization With Unobserved Preference Heterogeneity
RLHF has emerged as a pivotal step in aligning language models with human objectives and values. It typically involves learning a reward model from human preference data and then using reinforcement learning to update th…
Ensemble Learningreinforcement-learningReinforcement Learning