How to Train Your Deep Research Agent? Prompt, Reward, and Policy Optimization in Search-R1

2026-02-23 · Yinuo Xu, Shuo Lu, Jianjie Cheng, Meng Wang, Qianlong Xie, Xingxing Wang, Ran He, Jian Liang arxiv

Deep Research agents tackle knowledge-intensive tasks through multi-round retrieval and decision-oriented generation. While reinforcement learning (RL) has been shown to improve performance in this paradigm, its contributions remain underexplored. To fully understand the role of RL, we conduct a systematic study along three decoupled dimensions: prompt template, reward function, and policy optimization. Our study reveals that: 1) the Fast Thinking template yields greater stability and better performance than the Slow Thinking template used in prior work; 2) the F1-based reward underperforms the EM due to training collapse driven by answer avoidance; this can be mitigated by incorporating action-level penalties, ultimately surpassing EM; 3) REINFORCE outperforms PPO while requiring fewer search actions, whereas GRPO shows the poorest stability among policy optimization methods. Building on these insights, we then introduce Search-R1++, a strong baseline that improves the performance of Search-R1 from 0.403 to 0.442 (Qwen2.5-7B) and 0.289 to 0.331 (Qwen2.5-3B). We hope that our findings can pave the way for more principled and reliable RL training strategies in Deep Research systems.

📄 PDF Abstract BibTeX arXiv:2602.19526

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

Reward Collapse in Aligning Large Language Models

2023-05-28 · Ziang Song, Tianle Cai, Jason D. Lee, Weijie J. Su

The extraordinary capabilities of large language models (LLMs) such as ChatGPT and GPT-4 are in part unleashed by aligning them with reward models that are trained on human preferences, which are often represented as ran…

Zero Shot Coordination for Sparse Reward Tasks with Diverse Reward Shapings

2026-04-28 · Keenan Powell, Peihong Yu, Pratap Tokekar arxiv

Many Multi-Agent Reinforcement Learning (MARL) agents fail to adapt properly to cooperating with agents trained with the same objectives but different seeds, algorithms, or other training differences. This is the problem…

Multi-agent Reinforcement Learning

Sample Efficiency in Sparse Reinforcement Learning: Or Your Money Back

2020-08-28 · Trevor A. McInroe

Sparse rewards present a difficult problem in reinforcement learning and may be inevitable in certain domains with complex dynamics such as real-world robotics. Hindsight Experience Replay (HER) is a recent replay memory…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Arena: a toolkit for Multi-Agent Reinforcement Learning

2019-07-20 · Qing Wang, Jiechao Xiong, Lei Han, Meng Fang 외

We introduce Arena, a toolkit for multi-agent reinforcement learning (MARL) research. In MARL, it usually requires customizing observations, rewards and actions for each agent, changing cooperative-competitive agent-inte…

Multi-agent Reinforcement Learning OpenAI Gym reinforcement-learning Reinforcement Learning+3

Know Your Agent: Reconnaissance-Driven Pentesting of AI Agents

2026-07-22 · Or Zion Eliav, Eyal Lenga, Shir Bernstien, Yisroel Mirsky arxiv

Traditional pentesting uses reconnaissance at each step to uncover unseen weaknesses, build stronger attacks, and advance the objective; we argue that AI agents require the same treatment. We formalize agent reconnaissan…