ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learning
To address the limited capability expansion and low sample efficiency of Reinforcement Learning (RL), recent methods have integrated ''hints'' into post-training, which are prefix segments of complete reasoning trajectories, aiming for powerful knowledge expansion and reasoning generalization. However, existing hint-based RL methods often neglect the role of difficulty in the hint-ratio schedule and relative-advantage estimation, resulting in unstable learning and excessive imitation of off-policy hints. To address this, we propose ADHint, which explicitly integrates difficulty into both processes to achieve a better trade-off between exploration and imitation. Specifically, we propose Adaptive Hint with Sample Difficulty Prior, which evaluates the difficulty of each sample under the current policy to schedule an appropriate hint ratio for rollout generation. Furthermore, we introduce Consistency-based Gradient Modulation alongside Selective Masking for Hint Preservation, which jointly modulate token-level gradients within hints to prevent biased and destructive updates. Additionally, we propose Advantage Estimation with Rollout Difficulty Posterior, which leverages the relative difficulty of rollouts with and without hints to compute their respective advantages, yielding more balanced updates. Extensive experiments across diverse modalities, model scales, model families, and domains demonstrate that ADHint achieves superior reasoning capabilities and out-of-distribution generalization. Code and datasets will be made publicly available upon paper acceptance.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement LearningSimilar Papers 제목 키워드 기반
Unlocking the Potential of Difficulty Prior in RL-based Multimodal Reasoning
In this work, we investigate how explicitly modeling problem's difficulty prior information shapes the effectiveness of reinforcement learning based fine-tuning for multimodal reasoning. Our exploration mainly comprises …
2kMathematical ReasoningMultimodal ReasoningPlacing Puzzle Pieces Where They Matter: A Question Augmentation Framework for Reinforcement Learning
Reinforcement learning has become a powerful approach for enhancing large language model reasoning, but faces a fundamental dilemma: training on easy problems can cause overfitting and pass@k degradation, while training …
Reinforcement LearningMathematical ReasoningLAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Models
Spatial reasoning is a cornerstone capability for intelligent systems to perceive and interact with the physical world. However, multimodal large language models (MLLMs) frequently suffer from hallucinations and imprecis…
Spatial ReasoningHi-TTRL: Regulating Consensus with Hints for Test-Time Reinforcement Learning
Test-time reinforcement learning (TTRL) improves the reasoning capabilities of large language models without labeled data by updating the policy with pseudo-labels constructed through majority voting. While effective, th…
Reinforcement LearningSelf-Hinting Language Models Enhance Reinforcement Learning
Group Relative Policy Optimization (GRPO) has recently emerged as a practical recipe for aligning large language models with verifiable objectives. However, under sparse terminal rewards, GRPO often stalls because rollou…
Reinforcement Learning