paper-with-me

홈 › Papers

Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards

2023-06-07 · NeurIPS 2023 11 · Alexandre Ramé, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean-Baptiste Gaya, Laure Soulier, Matthieu Cord

Foundation models are first pre-trained on vast unsupervised datasets and then fine-tuned on labeled data. Reinforcement learning, notably from human feedback (RLHF), can further align the network with the intended usage. Yet the imperfections in the proxy reward may hinder the training and lead to suboptimal results; the diversity of objectives in real-world tasks and human opinions exacerbate the issue. This paper proposes embracing the heterogeneity of diverse rewards by following a multi-policy strategy. Rather than focusing on a single a priori reward, we aim for Pareto-optimal generalization across the entire space of preferences. To this end, we propose rewarded soup, first specializing multiple networks independently (one for each proxy reward) and then interpolating their weights linearly. This succeeds empirically because we show that the weights remain linearly connected when fine-tuned on diverse rewards from a shared pre-trained initialization. We demonstrate the effectiveness of our approach for text-to-text (summarization, Q&A, helpful assistant, review), text-image (image captioning, text-to-image generation, visual grounding, VQA), and control (locomotion) tasks. We hope to enhance the alignment of deep models, and how they interact with the world in all its diversity.

📄 PDF Abstract BibTeX arXiv:2306.04488

Code (1)

alexrame/rewardedsoups 공식 구현 pytorch

Tasks

DiversityImage CaptioningImage GenerationText SummarizationText to Image GenerationText-to-Image GenerationVisual GroundingVisual Question Answering (VQA)

Similar Papers 제목 키워드 기반

UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality

2025-03-10 · Zelei Cheng, Xin-Qiang Cai, Yuting Tang, Pushi Zhang 외

Reinforcement Learning from Human Feedback (RLHF) has become a cornerstone for aligning large language models (LLMs) with human values. However, existing approaches struggle to capture the multi-dimensional, distribution…

Bone Soups: A Seek-and-Soup Model Merging Approach for Controllable Multi-Objective Generation

2025-02-15 · Guofu Xie, Xiao Zhang, Ting Yao, Yunsheng Shi

User information needs are often highly diverse and varied. A key challenge in current research is how to achieve controllable multi-objective generation while enabling rapid adaptation to accommodate diverse user demand…

Multi-Objective Reinforcement Learning

Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging

2025-10-20 · Tiancheng Hu, Benjamin Minixhofer, Nigel Collier arxiv

The "alignment tax" of post-training is typically framed as a drop in task accuracy. We show it also involves a severe loss of calibration, making models overconfident, less reliable, and model outputs less diverse. We s…

Diversity-Rewarded CFG Distillation

2024-10-08 · Geoffrey Cideron, Andrea Agostinelli, Johan Ferret, Sertan Girgin 외

Generative models are transforming creative domains such as music generation, with inference-time strategies like Classifier-Free Guidance (CFG) playing a crucial role. However, CFG doubles inference cost while limiting …

DiversityMusic Generation

WARP: On the Benefits of Weight Averaged Rewarded Policies

2024-06-24 · Alexandre Ramé, Johan Ferret, Nino Vieillard, Robert Dadashi 외

Reinforcement learning from human feedback (RLHF) aligns large language models (LLMs) by encouraging their generations to have high rewards, using a reward model trained on human preferences. To prevent the forgetting of…