ProDS: Preference-oriented Data Selection for Instruction Tuning
Instruction data selection aims to identify a high-quality subset from the training set that matches or exceeds the performance of the full dataset on target tasks. Existing methods focus on the instruction-to-response mapping, but neglect the human preference for diverse responses. In this paper, we propose Preference-oriented Data Selection method (ProDS) that scores training samples based on their alignment with preferences observed in the target set. Our key innovation lies in shifting the data selection criteria from merely estimating features for accurate response generation to explicitly aligning training samples with human preferences in target tasks. Specifically, direct preference optimization (DPO) is employed to estimate human preferences across diverse responses. Besides, a bidirectional preference synthesis strategy is designed to score training samples according to both positive preferences and negative preferences. Extensive experimental results demonstrate our superiority to existing task-agnostic and targeted methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Response GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuning
Instruction tuning has underscored the significant potential of large language models (LLMs) in producing more human-controllable and effective outputs in various domains. In this work, we focus on the data selection pro…
Instruction-Oriented Preference Alignment for Enhancing Multi-Modal Comprehension Capability of MLLMs
Preference alignment has emerged as an effective strategy to enhance the performance of Multimodal Large Language Models (MLLMs) following supervised fine-tuning. While existing preference alignment methods predominantly…
HallucinationHallucination EvaluationQuestion AnsweringVisual Question AnsweringStyle over Story: Measuring LLM Narrative Preferences via Structured Selection
We introduce a constraint-selection-based experiment design for measuring narrative preferences of Large Language Models (LLMs). This design offers an interpretable lens on LLMs' narrative selection behavior. We develope…
PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation
Reward models are pivotal for aligning Large Language Models (LLMs) with human preferences. Existing approaches face two key limitations: Discriminative reward models require large-scale annotated data, as they cannot ex…
Biomedical Visual Instruction Tuning with Clinician Preference Alignment
Recent advancements in multimodal foundation models have showcased impressive capabilities in understanding and reasoning with visual and textual information. Adapting these foundation models trained for general usage to…
Instruction FollowingVisual Question Answering (VQA)