SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling

2024-05-21 · Xingzhou Lou, Junge Zhang, Jian Xie, Lifeng Liu, Dong Yan, Kaiqi Huang

Human preference alignment is critical in building powerful and reliable large language models (LLMs). However, current methods either ignore the multi-dimensionality of human preferences (e.g. helpfulness and harmlessness) or struggle with the complexity of managing multiple reward models. To address these issues, we propose Sequential Preference Optimization (SPO), a method that sequentially fine-tunes LLMs to align with multiple dimensions of human preferences. SPO avoids explicit reward modeling, directly optimizing the models to align with nuanced human preferences. We theoretically derive closed-form optimal SPO policy and loss function. Gradient analysis is conducted to show how SPO manages to fine-tune the LLMs while maintaining alignment on previously optimized dimensions. Empirical results on LLMs of different size and multiple evaluation datasets demonstrate that SPO successfully aligns LLMs across multiple dimensions of human preferences and significantly outperforms the baselines.

📄 PDF Abstract BibTeX arXiv:2405.12739

Code (2)

hiyouga/llama-factory 공식 구현 pytorch

MindSpore-scientific/code-14/tree/main/Three-Dimensional-Lip-Motion-Network-for-Text-Independent-Speaker-Recognition-master mindspore

Methods 이 논문이 사용한 방법론

ALIGN In the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss…

LifeAlign: Lifelong Alignment for Large Language Models with Memory-Augmented Focalized Preference Optimization

2025-09-21 · Junsong Li, Jie Zhou, Bihao Zhan, Yutao Yang 외 arxiv

Alignment plays a crucial role in Large Language Models (LLMs) in aligning with human preferences on a specific task/domain. Traditional alignment methods suffer from catastrophic forgetting, where models lose previously…

Dimensionality Reduction

2D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervision

2024-10-25 · Shilong Li, Yancheng He, Hui Huang, Xingyuan Bu 외

Recent advancements in Direct Preference Optimization (DPO) have significantly enhanced the alignment of Large Language Models (LLMs) with human preferences, owing to its simplicity and effectiveness. However, existing m…

Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansion

2026-05-12 · ShiYing Huang, Liang Lin, Yuer Li, Kaiwen Luo 외 arxiv

In the realm of multi-objective alignment for large language models, balancing disparate human preferences often manifests as a zero-sum conflict. Specifically, the intrinsic tension between competing goals dictates that…

PREFINE: Preference-Based Implicit Reward and Cost Fine-Tuning for Safety Alignment

2026-05-20 · Richa Verma, Bavish Kulur, Sanjay Chawla, Balaraman Ravindran arxiv

We address the problem of making a pre-trained reinforcement learning (RL) policy safety-aware by incorporating cost constraints without retraining it from scratch. While costs could be numerically encoded, we assume a m…

Computational Efficiency Reinforcement Learning Continuous Control Decision Making

Learning from Noisy Preferences: A Semi-Supervised Learning Approach to Direct Preference Optimization

2026-04-27 · Xinxin Liu, Ming Li, Zonglin Lyu, Yuzhang Shang 외 arxiv

Human visual preferences are inherently multi-dimensional, encompassing aesthetics, detail fidelity, and semantic alignment. However, existing datasets provide only single, holistic annotations, resulting in severe label…