Optimal Control-Based Baseline for Guided Exploration in Policy Gradient Methods
In this paper, a novel optimal control-based baseline function is presented for the policy gradient method in deep reinforcement learning (RL). The baseline is obtained by computing the value function of an optimal control problem, which is formed to be closely associated with the RL task. In contrast to the traditional baseline aimed at variance reduction of policy gradient estimates, our work utilizes the optimal control value function to introduce a novel aspect to the role of baseline -- providing guided exploration during policy learning. This aspect is less discussed in prior works. We validate our baseline on robot learning tasks, showing its effectiveness in guided exploration, particularly in sparse reward environments.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep Reinforcement LearningPolicy Gradient MethodsReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Learning Dexterous Manipulation from Suboptimal Experts
Learning dexterous manipulation in high-dimensional state-action spaces is an important open challenge with exploration presenting a major bottleneck. Although in many cases the learning process could be guided by demons…
Offline RLQ-LearningControllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning
Reinforcement Learning with verifiable rewards (RLVR) has emerged as a primary learning paradigm for enhancing the reasoning capabilities of multi-modal large language models (MLLMs). However, during RL training, the eno…
Reinforcement LearningPNS: Population-Guided Novelty Search for Reinforcement Learning in Hard Exploration Environments
Reinforcement Learning (RL) has made remarkable achievements, but it still suffers from inadequate exploration strategies, sparse reward signals, and deceptive reward functions. To alleviate these problems, a Population-…
continuous-controlContinuous Controlreinforcement-learningReinforcement Learning+1Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flow
Controlling high-dimensional systems in biological and robotic applications is challenging due to expansive state-action spaces, where effective exploration is critical. Commonly used exploration strategies in reinforcem…
Dimensionality ReductionReinforcement LearningContinuous ControlHamiltonian Policy Optimization
Approximating optimal policies in reinforcement learning (RL) is often necessary in many real-world scenarios, which is termed as policy optimization. By viewing the reinforcement learning from the perspective of variati…
continuous-controlContinuous ControlMuJoCoreinforcement-learning+2