HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents

2026-05-18 · Woongyeng Yeo, Yumin Choi, Taekyung Ki, Sung Ju Hwang arxiv

Training long-horizon LLM agents with reinforcement learning is challenging because sparse outcome rewards reveal whether a task succeeds, but not which intermediate actions caused the outcome or how they should be corrected. Recent methods alleviate this issue by generating rewards or textual hints from turn-level action-output signals, or by using feedback-conditioned self-distillation. However, generating feedback at every turn is inefficient when many intermediate turns are already successful or neutral, and applying feedback at a fixed or misaligned turn often fails to supervise the actions that contributed to the failure. To bridge this gap, we propose HINT-SD, a targeted self-distillation framework that uses full-trajectory hindsight to select failure-relevant actions and applies feedback-conditioned distillation only to targeted action spans. Experiments on BFCL v3 and AppWorld show that our method outperforms the dense per-turn feedback baseline by up to 13.60 percentage points on average while achieving a 2.26$\times$ reduction in time per training step, suggesting that selecting where to distill is key to effective and efficient long-horizon agent training.

📄 PDF Abstract BibTeX arXiv:2605.17873

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

Hindsight Hint Distillation: Scaffolded Reasoning for SWE Agents from CoT-free Answers

2026-05-12 · Shengjie Wang, Guanghe Li, Zonghan Yang, Yang Gao arxiv

Solving complex long-horizon tasks requires strong planning and reasoning capabilities. Although datasets with explicit chain-of-thought (CoT) rationales can substantially benefit learning, they are costly to obtain. To …

HEAL: Hindsight Entropy-Assisted Learning for Reasoning Distillation

2026-03-11 · Wenjing Zhang, Jiangze Yan, Jieyun Huang, Yi Shen 외 arxiv

Distilling reasoning capabilities from Large Reasoning Models (LRMs) into smaller models is typically constrained by the limitation of rejection sampling. Standard methods treat the teacher as a static filter, discarding…

TurnSight: Turn-Level Hindsight Self-Distillation for Tool-Integrated Reasoning

2026-08-04 · Changle Qu, Sunhao Dai, Hengyi Cai, Yuqi Zhou 외 hf

Tool-Integrated Reasoning (TIR) enables LLMs to solve complex tasks through iterative tool interactions. However, existing reinforcement learning methods often rely on trajectory-level supervision, limiting fine-grained …

Reinforcement Learning

H^2SD: Hybrid Hindsight Self-Distillation

2026-07-21 · Qiye Cai, Yichuan Ma, Linyang Li, Peiji Li 외 arxiv

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models on tasks such as mathematical reasoning and code generation. However, most RLVR methods…

Reinforcement Learning Mathematical Reasoning Code Generation

From Hindsight to Foresight: Self-Encouraged Hindsight Distillation for Knowledge-based Visual Question Answering

2025-11-14 · Yu Zhao, Ying Zhang, Xuhui Sui, Baohang Zhou 외 arxiv

Knowledge-based Visual Question Answering (KBVQA) necessitates external knowledge incorporation beyond cross-modal understanding. Existing KBVQA methods either utilize implicit knowledge in multimodal large language mode…

Visual Question Answering