Rethinking the Design of Reinforcement Learning-Based Deep Research Agents
Large language models (LLMs) augmented with external tools are increasingly deployed as deep research agents that gather, reason over, and synthesize web information to answer complex queries. Although recent open-source systems achieve strong empirical performance via reinforcement learning from web interactions, the impact of key design choices remains under-explored. We formalize deep research as reinforcement learning in an episodic finite Markov decision process and construct a competitive baseline agent grounded in this formulation. Building on this foundation, we systematically examine critical design decisions at both training and inference time and identify four factors that substantially improve performance: replacing rule-based rewards with AI feedback from an LLM judge, fine-tuning with the on-policy RLOO algorithm instead of the off-policy GRPO algorithm, filtering low-quality training samples, and employing an error-tolerant test-time rollout strategy. Together, these design choices yield a deep research agent that establishes state-of-the-art performance among 7B-scale agents when evaluated across ten widely used benchmarks.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement LearningSimilar Papers 제목 키워드 기반
Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue Systems
Dialogue policy learning for task-oriented dialogue systems has enjoyed great progress recently mostly through employing reinforcement learning methods. However, these approaches have become very sophisticated. It is tim…
DecoderMulti-Label ClassificationMUlTI-LABEL-ClASSIFICATIONreinforcement-learning+3Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models
Defining action spaces for conversational agents and optimizing their decision-making process with reinforcement learning is an enduring challenge. Common practice has been to use handcrafted dialog acts, or the output v…
Decision MakingDialogue GenerationDialogue ManagementGoal-Oriented Dialogue Systems+3The Tool Illusion: Rethinking Tool Use in Web Agents
As web agents rapidly evolve, an increasing body of work has moved beyond conventional atomic browser interactions and explored tool use as a higher-level action paradigm. Although prior studies have shown the promise of…
Rethinking the Foundations for Continual Reinforcement Learning
Algorithms and approaches for continual reinforcement learning have gained increasing attention. Much of this early progress rests on the foundations and standard practices of traditional reinforcement learning, without …
Continual Learningreinforcement-learningReinforcement LearningRethinking Agentic Reinforcement Learning In Large Language Models
Reinforcement Learning (RL) has traditionally focused on training specialized agents to optimize predefined reward functions within narrowly defined environments. However, the advent of powerful Large Language Models (LL…
Reinforcement Learning