MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization
Multi-negative preference optimization under the Plackett--Luce (PL) model extends Direct Preference Optimization (DPO) by leveraging comparative signals across one preferred and multiple rejected responses. However, optimizing over large negative pools is costly, and many candidates contribute redundant gradients due to their similar effects on policy updates. We introduce MASS-DPO, a multi-negative active sample selection method that derives a PL-specific Fisher-information objective for selecting compact, informative negative subsets within each prompt. The resulting log-determinant objective selects negatives that contribute complementary information for policy updates, yielding compact subsets that retain the full pool's information while reducing redundancy. In practice, this favors negatives whose gradients cover different update directions, reducing redundant signal from near-duplicate candidates while preserving the most useful training information. Across four benchmarks spanning recommendation and multiple-choice QA and three model families, MASS-DPO consistently exceeds or matches existing methods in accuracy, improves Recall/NDCG and margin-based optimization dynamics, and delivers stronger alignment with substantially fewer negatives.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering
Active learning promises to alleviate the massive data needs of supervised machine learning: it has successfully improved sample efficiency by an order of magnitude on traditional tasks like topic classification and obje…
Active LearningObject RecognitionQuestion AnsweringTopic Classification+2Activation-Based Active Learning for In-Context Learning: Challenges and Insights
Deep active learning has previously been explored for LLM in-context sample selection, but not with methods that utilise recent advances in understanding of transformer activations. In this paper, we test the hypothesis …
Active LearningFunnelAL: Retrieve-then-Rank Active Learning for Single-Class Discovery
We present FunnelAL, a retrieve-then-rank active learning system for single-class discovery, which adapts the multi-stage funnel architecture of industrial recommender systems to data annotation. Large-scale supervised l…
Image ClassificationActive LearningActive entailment encoding for explanation tree construction using parsimonious generation of hard negatives
Entailment trees have been proposed to simulate the human reasoning process of explanation generation in the context of open--domain textual question answering. However, in practice, manually constructing these explanati…
Explanation GenerationQuestion AnsweringIterative Loop Method Combining Active and Semi-Supervised Learning for Domain Adaptive Semantic Segmentation
Semantic segmentation is an important technique for environment perception in intelligent transportation systems. With the rapid development of convolutional neural networks (CNNs), road scene analysis can usually achiev…
Active LearningDomain AdaptationSemantic SegmentationSemi-Supervised Semantic Segmentation