On the Position Bias of On-Policy Distillation

2026-06-21 · Yan Xie, Sijie Zhu, Tiansheng Wen, Bo Chen, Yifei Wang arxiv

On-Policy Distillation (OPD) improves the learning efficiency of standard reinforcement learning through dense, token-level supervision from teachers. In the standard KL objective of OPD, token-level losses are uniformly averaged, implying equal weights for all tokens. However, we discover that not all tokens are created equal: as student rollouts grow longer, they deviate further from the teacher's distribution, leading to degraded supervision quality at later positions. As a result, OPD using only the first 30% of tokens can perform comparably to using all tokens, whereas OPD using only the last 30% of tokens barely learns anything. In this work, we provide a principled understanding of this issue through the lens of constrained optimization. Based on these insights, we derive Importance-Weighted On-Policy Distillation (IW-OPD), in which the weight assigned to each token depends on the accumulated discrepancy between the student's and teacher's distributions, naturally upweighting earlier tokens and downweighting later ones with larger deviations. We show that IW-OPD converges significantly faster than OPD, with better learning efficiency, and achieves better final performance than standard OPD in both same-size and cross-scale settings, improving performance up to 6.9 points on AIME-2025.

📄 PDF Abstract BibTeX arXiv:2606.22600

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

Contrastive Reinforced Policy Optimization via Privileged Self-Distillation

2026-07-30 · Xingjian Wu, Junlin Liu, Xingchen Liu, Xuhang Zhu 외 arxiv

Recent advances in post-training Large Language Models (LLMs) increasingly rely on Reinforcement Learning with Verifiable Rewards (RLVR) or On-Policy Self-Distillation (OPSD). While OPSD provides dense, logit-level super…

Reinforcement Learning Contrastive Learning

Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation

2026-05-19 · Bing Wang, Shaotian Yan, Chen Shen, kaiyuan liu 외 arxiv

Large language models (LLMs) have achieved remarkable success in complex reasoning tasks via long chain-of-thought (CoT), yet their immense computational overhead hinders real-world deployment. LLM reasoning distillation…

I-SDPO: Instance-Level Adaptive Self-Distillation Policy Optimization

2026-08-13 · Yubo Zhang, Xinhong Ma, Zezhong Tan, Ziqiang Dong arxiv

Group Relative Policy Optimization (GRPO) learns from reward differences within a rollout group, but receives no useful relative signal when every sampled response is incorrect. Privileged self-distillation can fill this…

dOPSD: On-Policy Self-Distillation for Diffusion Language Models

2026-07-05 · Phuong Tuan Dat, Qi Li, Xinchao Wang hf

Diffusion large language models (dLLMs) generate text by iteratively denoising a masked sequence, offering a parallel alternative to autoregressive models, but eliciting strong reasoning through post-training remains dif…

Reinforcement Learning Code Generation

On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity

2026-06-24 · Andrei Liviu Nicolicioiu, Mohammad Pezeshki, Aaron Courville arxiv

On-policy self-distillation achieves strong pass@1 accuracy by using a single model as both teacher and student, with the teacher conditioned on a correct demonstration to provide dense token-level feedback. We show that…