Active Learning for Direct Preference Optimization
Direct preference optimization (DPO) is a form of reinforcement learning from human feedback (RLHF) where the policy is learned directly from preferential feedback. Although many models of human preferences exist, the critical task of selecting the most informative feedback for training them is under-explored. We propose an active learning framework for DPO, which can be applied to collect human feedback online or to choose the most informative subset of already collected feedback offline. We propose efficient algorithms for both settings. The key idea is to linearize the DPO objective at the last layer of the neural network representation of the optimized policy and then compute the D-optimal design to collect preferential feedback. We prove that the errors in our DPO logit estimates diminish with more feedback. We show the effectiveness of our algorithms empirically in the setting that matches our theory and also on large language models.
Code (0)
등록된 구현이 없습니다.
Tasks
Active LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Bayesian Preference Elicitation: Human-In-The-Loop Optimization of An Active Prosthesis
Tuning active prostheses for people with amputation is time-consuming and relies on metrics that may not fully reflect user needs. We introduce a human-in-the-loop optimization (HILO) approach that leverages direct user …
Direct Preference-Based Evolutionary Multi-Objective Optimization with Dueling Bandit
Optimization problems find widespread use in both single-objective and multi-objective scenarios. In practical applications, users aspire for solutions that converge to the region of interest (ROI) along the Pareto front…
Evolutionary AlgorithmsProtein Structure PredictionClinical Reading Comprehension with Encoder-Decoder Models Enhanced by Direct Preference Optimization
Extractive question answering over clinical text is a crucial need to help deal with the deluge of clinical text generated in hospitals. While encoder models (e.g., BERT) have been popular for this reading comprehension …
DecoderExtractive Question-AnsweringQuestion AnsweringReading ComprehensionStatistical Rejection Sampling Improves Preference Optimization
Improving the alignment of language models with human preferences remains an active research challenge. Previous approaches have primarily utilized Reinforcement Learning from Human Feedback (RLHF) via online RL methods …
Language ModellingLarge Language ModelActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignment
The recent success of using human preferences to align large language models (LLMs) has significantly improved their performance in various downstream tasks like question answering, mathematical reasoning, and code gener…
Code GenerationMathematical ReasoningQuestion Answering