PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization
Direct Preference Optimization (DPO) simplifies alignment through pairwise comparisons but assumes all observed preferences are reliable. Real data often violates this assumption, leading to reversed, weak, or ambiguous labels that cause harmful policy updates. To address this, we propose Posterior Label Correction DPO (PLC-DPO) to robustly optimize preferences by routing each pair's training signal as a clean, flip, or tie case. The key idea is to use the calibrated policy-reference margin as online evidence to take appropriate correction actions. This reframes noisy preference learning as actively correcting supervision direction and strength rather than merely filtering suspicious examples. Across 57 dataset-model-benchmark cells, PLC-DPO obtains the best mean win rate against DPO (60.5 vs. 55.5 for the next-best method). Injected-noise and tie stress tests, human disagreement analysis, and self-confirmation diagnostics further show that the routing remains stable and distinguishes flipped from weakly directional pairs.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Robust Self-Training with Closed-loop Label Correction for Learning from Noisy Labels
Training deep neural networks with noisy labels remains a significant challenge, often leading to degraded performance. Existing methods for handling label noise typically rely on either transition matrix, noise detectio…
Bilevel OptimizationAddressing Ambiguity of Emotion Labels Through Meta-Learning
Emotion labels in emotion recognition corpora are highly noisy and ambiguous, due to the annotators' subjective perception of emotions. Such ambiguity may introduce errors in automatic classification and affect the overa…
Emotion RecognitionMeta-LearningLabel Correction of Crowdsourced Noisy Annotations with an Instance-Dependent Noise Transition Model
The predictive ability of supervised learning algorithms hinges on the quality of annotated examples, whose labels often come from multiple crowdsourced annotators with diverse expertise. To aggregate noisy crowdsourced …
Uncertainty-Penalized Direct Preference Optimization
Aligning Large Language Models (LLMs) to human preferences in content, style, and presentation is challenging, in part because preferences are varied, context-dependent, and sometimes inherently ambiguous. While successf…
reinforcement-learningReinforcement LearningRepFace: Refining Closed-Set Noise with Progressive Label Correction for Face Recognition
Face recognition has made remarkable strides, driven by the expanding scale of datasets, advancements in various backbone and discriminative losses. However, face recognition performance is heavily affected by the label …
Face Recognition