CAE: Repurposing the Critic as an Explorer in Deep Reinforcement Learning

2025-03-23 · Yexin Li, Pring Wong, Hanfang Zhang, Shuo Chen, Siyuan Qi

Exploration remains a critical challenge in reinforcement learning, as many existing methods either lack theoretical guarantees or fall short of practical effectiveness. In this paper, we introduce CAE, a lightweight algorithm that repurposes the value networks in standard deep RL algorithms to drive exploration without introducing additional parameters. CAE utilizes any linear multi-armed bandit technique and incorporates an appropriate scaling strategy, enabling efficient exploration with provable sub-linear regret bounds and practical stability. Notably, it is simple to implement, requiring only around 10 lines of code. In complex tasks where learning an effective value network proves challenging, we propose CAE+, an extension of CAE that incorporates an auxiliary network. This extension increases the parameter count by less than 1% while maintaining implementation simplicity, adding only about 10 additional lines of code. Experiments on MuJoCo and MiniHack show that both CAE and CAE+ outperform state-of-the-art baselines, bridging the gap between theoretical rigor and practical efficiency.

📄 PDF Abstract BibTeX arXiv:2503.18980

Code (0)

등록된 구현이 없습니다.

Tasks

Deep Reinforcement LearningEfficient ExplorationMuJoCo

L2Explorer: A Lifelong Reinforcement Learning Assessment Environment

2022-03-14 · Erik C. Johnson, Eric Q. Nguyen, Blake Schreurs, Chigozie S. Ewulum 외

Despite groundbreaking progress in reinforcement learning for robotics, gameplay, and other complex domains, major challenges remain in applying reinforcement learning to the evolving, open-world problems often found in …

Continual Learning Lifelong learning reinforcement-learning Reinforcement Learning+2

OVD-Explorer: A General Information-theoretic Exploration Approach for Reinforcement Learning

2021-09-29 · Jinyi Liu, Zhi Wang, Yan Zheng, Jianye Hao 외

Many exploration strategies are built upon the optimism in the face of the uncertainty (OFU) principle for reinforcement learning. However, without considering the aleatoric uncertainty, existing methods may over-explore…

MuJoCo reinforcement-learning Reinforcement Learning (RL)

Toward Debugging Deep Reinforcement Learning Programs with RLExplorer

2024-10-06 · Rached Bouchoucha, Ahmed Haj Yahmed, Darshan Patil, Janarthanan Rajendran 외

Deep reinforcement learning (DRL) has shown success in diverse domains such as robotics, computer games, and recommendation systems. However, like any other software system, DRL-based software systems are susceptible to …

Deep Reinforcement Learning Fault Diagnosis Recommendation Systems reinforcement-learning+1

LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models

2025-05-21 · Qianyue Hao, Yiwen Song, Qingmin Liao, Jian Yuan 외

Policy exploration is critical in reinforcement learning (RL), where existing approaches include greedy, Gaussian process, etc. However, these approaches utilize preset stochastic processes and are indiscriminately appli…

MuJoCo Reinforcement Learning (RL)

EXPLORER: Exploration-guided Reasoning for Textual Reinforcement Learning

2024-03-15 · Kinjal Basu, Keerthiram Murugesan, Subhajit Chaudhury, Murray Campbell 외

Text-based games (TBGs) have emerged as an important collection of NLP tasks, requiring reinforcement learning (RL) agents to combine natural language understanding with reasoning. A key challenge for agents attempting t…

Natural Language Understanding reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1