TIER: Trajectory-Invariant Execution Rewards for Multi-Step Tool Composition

2026-05-16 · Anay Kulkarni, ChiaEn Lu, Dheeraj Mekala, Jayanth Srinivasa, Gaowen Liu, Jingbo Shang arxiv

Tool use enables large language models to solve complex tasks through sequences of API calls, yet existing reinforcement learning approaches fail to scale to multi-step composition settings. Outcome-based rewards provide only sparse feedback, while trajectory-supervised rewards depend on annotated reference solutions, penalizing valid alternatives and limiting scalability. We propose TIER: Trajectory-Invariant Execution Rewards, a reward framework that derives supervision directly from function schemas and runtime execution, rather than from reference trajectories. The reward decomposes into format validity, schema adherence, execution success, and answer correctness, providing dense, interpretable sequence-level feedback derived from fine-grained verification of individual steps of tool use. This design allows any valid execution path to receive credit, naturally supporting multiple solution strategies and adapting to evolving tool interfaces. On DepthBench, a compositional benchmark stratified by depth (1 to 6 steps), TIER achieves >90% accuracy across steps, where trajectory-supervised rewards collapse beyond step-4. We further demonstrate consistent gains on benchmarks like BFCL v3 and NestFUL. Ablation studies confirm that all reward components are necessary, highlighting the importance of multi-level supervision for compositional reasoning.

📄 PDF Abstract BibTeX arXiv:2605.16790

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

CacheRL:Multi-Turn Tool-Calling Agents via Cached Rollouts and Hybrid Reward

2026-06-12 · Md Amirul Islam, Sumiran Thakur, Huancheng Chen, Su Min Park 외 arxiv

We present CacheRL, a system for training small agent foundation models that achieves 92 percent process accuracy on multi-step tool-calling tasks, approaching GPT-5's 94 percent while requiring 100 times less compute. O…

Reinforcement Learning

Signal Reshaping for GRPO in Weak-Feedback Agentic Code Repair

2026-05-08 · Jia Li, Yuxin Su, Ting Peng, Hailiang Huang 외 arxiv

Code-agent RL often receives weak feedback: rollout-time signals are reliable and executable, but capture only necessary or surface conditions for task success rather than the target semantic predicate. Using agentic com…

Code Repair

AeroCopilotBench: A Two-Tier Benchmark for Evaluating LLM Agents as Aviation Copilots in an Interactive Virtual Cockpit Environment

2026-08-17 · Yuchen Yuan, Zhenghuang Wu, Yuangan Li, Liang Ma 외 arxiv

Large language model (LLM) agents may assist flight crews with complex decisions and task execution, but existing aviation evaluations centered on static knowledge do not support systematic testing of procedural executio…

DynSTEER: Dynamic Stage-wise Trajectory Evaluation and Execution-time Review for Agents

2026-09-13 · Zhichao Shi, Xuhui Jiang, Wenjie Zhang, Xiaojun Wu 외 arxiv

Large language model agents are increasingly deployed for long-horizon task execution, raising a central granularity question for trajectory evaluation: whole-trajectory verification is too coarse to capture concrete fai…

WorldReward: Reward Modeling for Camera-Conditioned World Models

2026-09-03 · Yibin Wang, Zehan Wang, Junshu Tang, Zhimin Li 외 hf

Camera-conditioned world models generate interactive videos in which commanded actions should induce the expected scene changes while appearance, geometry, and temporal dynamics remain coherent. Existing rewards assess t…