홈 › Papers

S-SPPO: Semantic-Calibrated Self-Play Preference Optimization

2026-06-01 · Xiwen Chen, Wenhui Zhu, Jingjing Wang, Peijie Qiu, Zhipeng Wang, Huayu Li, ZhengXiao He, Xuanzhao Dong, Prayag Tiwari, Mingkun Xu, Yujian Xiong, Feng Luo, Abolfazl Razi, Brendan Hogan Rappazzo, Anderson Schneider, Yuriy Nevmyvaka arxiv

Aligning Large Language Models (LLMs) with human preferences is often formulated via Direct Preference Optimization (DPO). However, the standard Bradley-Terry instantiation of DPO is limited in modeling common departures from transitivity in human preferences. To address this, recent work has introduced Self-Play Preference Optimization (SPPO), which iteratively refines the policy by training on self-generated win-lose pairs. Our investigation, however, reveals a critical instability in SPPO: the optimization is prone to policy degeneration when the preference oracle assigns overly confident wins to semantically indistinguishable responses. To mitigate this, we propose S-SPPO, a dual-space semantic calibration framework comprising: i) Supervision Calibration via semantic gating, which anneals win rate targets toward the maximum-entropy baseline as semantic overlap increases; and ii) Representation Calibration via latent repulsion to enforce geometric diversity to prevent manifold collapse and maintain latent diversity between chosen and rejected samples. Theoretically, we show that the calibration preserves the constant-sum game structure, facilitating convergence to a Nash Equilibrium. Empirically, S-SPPO avoids the performance degradation seen in prior methods, achieving 52.19% win rate and 47.46% length-controlled win rate on AlpacaEval 2.0 with Llama-3-8B, without using additional human-annotated preferences during training. The code will be available at https://github.com/xiwenc1/s-sppo.

📄 PDF Abstract BibTeX arXiv:2606.01561

Code (0)

등록된 구현이 없습니다.

Self-Play Preference Optimization for Language Model Alignment

2024-05-01 · Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji 외

Standard reinforcement learning from human feedback (RLHF) approaches relying on parametric models like the Bradley-Terry model fall short in capturing the intransitivity and irrationality in human preferences. Recent ad…

Language Modeling Language Modelling model

Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic Large Language Model Alignment

2026-05-17 · Yucong Huang, Xiucheng Li, Kaiqi Zhao, Jing Li arxiv

Standard RLHF relies on transitive scalar rewards, failing to capture the cyclic nature of human preferences. While some approaches like the General Preference Model (GPM) address this, we identify a theoretical limitati…

Accelerated Preference Optimization for Large Language Model Alignment

2024-10-08 · Jiafan He, Huizhuo Yuan, Quanquan Gu

Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal tool for aligning large language models (LLMs) with human preferences. Direct Preference Optimization (DPO), one of the most popular approaches, …

Language Modeling Language Modelling Large Language Model

Symmetric Reinforcement Learning Loss for Robust Learning on Diverse Tasks and Model Scales

2024-05-27 · Ju-Seung Byun, Andrew Perrault

Reinforcement learning (RL) training is inherently unstable due to factors such as moving targets and high gradient variance. Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning from AI Feedback …

Atari Games MuJoCo reinforcement-learning Reinforcement Learning+1

SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks

2026-04-10 · Tianyi Wang, Yixia Li, Long Li, Yibiao Chen 외 arxiv

Proximal Policy Optimization (PPO) is central to aligning Large Language Models (LLMs) in reasoning tasks with verifiable rewards. However, standard token-level PPO struggles in this setting due to the instability of tem…