On-Policy Self-Distillation without Any Supervision

2026-08-06 · Yijiang Li, Bingyang Wang, Yijun Liang, Yunjie Tian, Di Fu, Nuno Vasconcelos arxiv

On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine "self"-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose unsupervised on-policy self-distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo solution by majority vote under a self-consistency threshold. It then conditions the model's distribution on the pseudo-solution and distills itself on the disagreeing completions, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT) such as OPSD and GRPO. On five mathematical reasoning benchmarks, i.e., AIME24, AIME25, HMMT25, MATH500, and AMC23, U-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at 4B and 8B scales, and outperforms OPSD by 3.2% and 2.3% on average, respectively. In thinking mode, U-OPSD stays on par with OPSD, ahead by 0.9% at 4B and level at 8B and surpassing GRPO by 0.7% and 1.1%, respectively. Code is available at https://github.com/williamium3000/u-opsd.

📄 PDF Abstract BibTeX arXiv:2608.06296

Code (0)

등록된 구현이 없습니다.

Tasks

Mathematical Reasoning

Self-Boosting Vision-Language Models with Noisy Student On-Policy Self-Distillation

2026-07-25 · Shuai Wang, Daoan Zhang, Zhe Tang, Hao Cheng 외 arxiv

Post-training enables vision-language models (VLMs) to understand human instructions and perform various downstream tasks. Current post-training methods usually rely on human-annotated data, distillation from external mo…

Reinforcement Learning Visual Reasoning

Neuron-Aware Data Selection for Annotation-Free LLM Self-Distillation

2026-07-02 · Zhuowei Chen, Xiang Lorraine Li arxiv

Post-training large language models (LLMs) without real-world interaction feedback or human-labeled supervision remains challenging, particularly in specialized domains where expert annotations are costly to obtain. Rece…

Domain Generalization Offline RL

Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoning

2026-05-21 · Hongbin Zhang, Chaozheng Wang, Kehai Chen, Youcheng Pan 외 arxiv

On-policy self-distillation (OPSD) is an emerging LLM post-training paradigm in which the model serves as its own teacher: conditioned on privileged information such as a reference trace or hint, the same policy provides…

Mathematical Reasoning

HPSD: Hybrid-Policy Self-Distillation for Text-Image-to-Video Diffusion Models

2026-08-13 · Jiazi Bu, Pengyang Ling, Yujie Zhou, Yibin Wang 외 arxiv

Text-Image-to-Video (TI2V) models are an emerging unified architecture, where a single model simultaneously supports text-to-video (T2V) and image-to-video (I2V) generation. Given a high-quality first frame or a detailed…

RISE: Recursive Improvement via Self-Extrapolating Policy Distillation

2026-09-04 · Yang Li, Semih Yavuz, Shafiq Joty arxiv

On-policy distillation (OPD) provides dense, per-token supervision for language model post-training, but its effectiveness is bottlenecked by teacher quality: external teachers suffer from distribution mismatch, while se…

Mathematical Reasoning Code Generation