ORPO-Distill: Mixed-Policy Preference Optimization for Cross-Architecture LLM Distillation
We introduce ORPO-Distill, a general-purpose method for cross-architecture LLM distillation that formulates the problem as a preference optimization task. Unlike standard CoT distillation, the approach transfers knowledge through diverse reasoning traces. It employs an Odds-Ratio Preference Optimization objective that contrasts teacher and student traces for more effective learning, and adopts a mixed-policy strategy for utilizing student-generated outputs, outperforming both off- and on-policy alternatives. Experiments on five datasets and multiple student models show consistent improvements over conventional black-box KD baselines.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Online Knowledge Distillation with Reward Guidance
This work studies knowledge distillation (KD) for large language models (LLMs) through preference optimization. We propose a reward-guided imitation learning framework for sequential KD, formulating a min-max optimizatio…
Imitation LearningKnowledge DistillationThe Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
On-policy distillation (OPD) and on-policy self-distillation (OPSD) have emerged as promising post-training methods for large language models, offering dense token-level supervision on trajectories sampled from the model…
Mathematical ReasoningLearning Loss Landscapes in Preference Optimization
We present an empirical study investigating how specific properties of preference datasets, such as mixed-quality or noisy data, affect the performance of Preference Optimization (PO) algorithms. Our experiments, conduct…
MuJoCoPreference as Reward, Maximum Preference Optimization with Importance Sampling
Preference learning is a key technology for aligning language models with human values. Reinforcement Learning from Human Feedback (RLHF) is a model-based algorithm to optimize preference learning, which first fits a rew…
KEPO: Knowledge-Enhanced Preference Optimization for Multimodal Reasoning with Applications to Medical VQA
Reinforcement learning (RL) has emerged as a promising paradigm for inducing explicit reasoning behaviors in large language and vision-language models. However, reasoning-oriented RL post-training remains fundamentally c…
Visual Question AnsweringReinforcement LearningMultimodal Reasoning