ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignment

2025-05-25 · Xiaoqiang Lin, Arun Verma, Zhongxiang Dai, Daniela Rus, See-Kiong Ng, Bryan Kian Hsiang Low

The recent success of using human preferences to align large language models (LLMs) has significantly improved their performance in various downstream tasks like question answering, mathematical reasoning, and code generation. However,3 achieving effective LLM alignment depends on high-quality human preference datasets. Collecting these datasets requires human preference annotation, which is costly and resource-intensive, necessitating efficient active data selection methods. Existing methods either lack a strong theoretical foundation or depend on restrictive reward function assumptions (e.g., linearity). To this end, we propose an algorithm, ActiveDPO, that uses a theoretically grounded data selection criterion for non-linear reward functions while directly leveraging the LLM itself to parameterize the reward model that is used for active data selection. As a result, ActiveDPO explicitly accounts for the influence of LLM on data selection, unlike methods that select the data without considering the LLM that is being aligned, thereby leading to more effective and efficient data collection. Extensive experiments show that ActiveDPO outperforms existing methods across various models and datasets.

📄 PDF Abstract BibTeX arXiv:2505.19241

Code (0)

등록된 구현이 없습니다.

Tasks

Code Generation Mathematical Reasoning Question Answering

Methods 이 논문이 사용한 방법론

ALIGN In the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss…

Statistical Rejection Sampling Improves Preference Optimization

2023-09-13 · Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman 외

Improving the alignment of language models with human preferences remains an active research challenge. Previous approaches have primarily utilized Reinforcement Learning from Human Feedback (RLHF) via online RL methods …

Language Modelling Large Language Model

Sample Efficient Preference Alignment in LLMs via Active Exploration

2023-12-01 · Viraj Mehta, Syrine Belakaria, Vikramjeet Das, Ojash Neopane 외

Preference-based feedback is important for many applications in machine learning where evaluation of a reward function is not feasible. Notable recent examples arise in preference alignment for large language models, inc…

reinforcement-learning Reinforcement Learning

MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization

2026-05-11 · Rohan Surana, Xintong Li, Sheldon Yu, Yiran Jenny Shen 외 arxiv

Multi-negative preference optimization under the Plackett--Luce (PL) model extends Direct Preference Optimization (DPO) by leveraging comparative signals across one preferred and multiple rejected responses. However, opt…

TSO: Self-Training with Scaled Preference Optimization

2024-08-31 · Kaihui Chen, Hao Yi, Qingyang Li, Tianyu Qi 외

Enhancing the conformity of large language models (LLMs) to human preferences remains an ongoing research challenge. Recently, offline approaches such as Direct Preference Optimization (DPO) have gained prominence as att…

Diversity

Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models

2026-05-06 · Huatian Zhang, Zhendong Mao, Lei Zhang, Yongdong Zhang arxiv

Direct Preference Optimization (DPO) has proven to be an effective solution for mitigating hallucination in Multimodal Large Language Models (MLLMs) by learning from preference pairs. One of its key challenges lies in ho…