Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoning
Distilling reasoning capabilities from strong to weak language models typically involves imitating specific solution trajectories, effectively transferring what to answer rather than how to reason. This trajectory-level imitation encourages memorization of instance-specific steps rather than acquisition of transferable problem-solving skills, limiting generalization to novel problems. We propose Strategy-Guided Policy Optimization (SGPO), which replaces instance-level trajectory imitation with reusable strategy distillation. SGPO extracts structured strategy descriptions from strong-model responses and, for each problem, constructs both autonomous and strategy-guided trajectories to enable direct comparison of the model's behavior with and without strategic guidance. The framework then addresses two key questions. For how to distill, a token-level forward-KL objective selectively transfers the distributional shift induced by strategy conditioning into the unguided policy, with proximal constraints ensuring stability. For when to distill, adaptive instance-level weighting strengthens guidance when autonomous exploration falls short and reduces it as the model's own competence grows. Experiments on four mathematical benchmarks across two model families show that SGPO consistently outperforms SFT, on-policy RL, and hybrid-policy baselines, improving the average score by 2.2 points over the strongest baseline on Qwen2.5-7B-Instruct. Analysis reveals that the forward-KL objective provides an inherently selective distillation signal that outperforms direct trajectory imitation, and that strategy distillation exhibits complementary scaling with base model capability.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Cross-Domain Energy-Guided Diffusion Generation for Off-Dynamics Reinforcement Learning
Off-dynamics offline reinforcement learning seeks to learn a target-domain policy from a large source dataset and a limited target dataset under mismatched transition dynamics. Existing approaches such as reward augmenta…
Reinforcement LearningOffline RLSketch-to-Skill: Bootstrapping Robot Learning with Human Drawn Trajectory Sketches
Training robotic manipulation policies traditionally requires numerous demonstrations and/or environmental rollouts. While recent Imitation Learning (IL) and Reinforcement Learning (RL) methods have reduced the number of…
Imitation Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)CACTO: Continuous Actor-Critic with Trajectory Optimization -- Towards global optimality
This paper presents a novel algorithm for the continuous control of dynamical systems that combines Trajectory Optimization (TO) and Reinforcement Learning (RL) in a single framework. The motivations behind this algorith…
continuous-controlContinuous ControlReinforcement Learning (RL)Beyond Where to Look: Trajectory-Guided Reinforcement Learning for Multimodal RLVR
Recent advances in Reinforcement Learning with Verifiable Rewards (RLVR) for multimodal large language models (MLLMs) have mainly focused on improving final answer correctness and strengthening visual grounding. However,…
Reinforcement LearningMultimodal ReasoningLogical ReasoningVisual GroundingPolicy-Based Trajectory Clustering in Offline Reinforcement Learning
We introduce a novel task of clustering trajectories from offline reinforcement learning (RL) datasets, where each cluster center represents the policy that generated its trajectories. By leveraging the connection betwee…
ClusteringD4RLOffline RLreinforcement-learning+4