Wasserstein Adaptive Value Estimation for Actor-Critic Reinforcement Learning
We present Wasserstein Adaptive Value Estimation for Actor-Critic (WAVE), an approach to enhance stability in deep reinforcement learning through adaptive Wasserstein regularization. Our method addresses the inherent instability of actor-critic algorithms by incorporating an adaptively weighted Wasserstein regularization term into the critic's loss function. We prove that WAVE achieves $\mathcal{O}\left(\frac{1}{k}\right)$ convergence rate for the critic's mean squared error and provide theoretical guarantees for stability through Wasserstein-based regularization. Using the Sinkhorn approximation for computational efficiency, our approach automatically adjusts the regularization based on the agent's performance. Theoretical analysis and experimental results demonstrate that WAVE achieves superior performance compared to standard actor-critic methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Computational EfficiencyDeep Reinforcement Learningreinforcement-learningReinforcement LearningSimilar Papers 제목 키워드 기반
Wasserstein Actor-Critic: Directed Exploration via Optimism for Continuous-Actions Control
Uncertainty quantification has been extensively used as a means to achieve efficient directed exploration in Reinforcement Learning (RL). However, state-of-the-art methods for continuous actions still suffer from high sa…
MuJoCoQ-LearningReinforcement Learning (RL)Uncertainty QuantificationActor-Critic learning for mean-field control in continuous time
We study policy gradient for mean-field control in continuous time in a reinforcement learning setting. By considering randomised policies with entropy regularisation, we derive a gradient expectation representation of t…
reinforcement-learningReinforcement Learning (RL)Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors
In reinforcement learning (RL), function approximation errors are known to easily lead to the Q-value overestimations, thus greatly reducing policy performance. This paper presents a distributional soft actor-critic (DSA…
continuous-controlContinuous ControlMuJoCoreinforcement-learning+2Deep Reinforcement Learning With Adaptive Combined Critics
The overestimation problem has long been popular in deep value learning, because function approximation errors may lead to amplified value estimates and suboptimal policies. There have been several methods to deal with t…
continuous-controlContinuous ControlDeep Reinforcement Learningreinforcement-learning+2Adaptive Ensemble Aggregation for Actor-Critics
Ensembles are ubiquitous in off-policy actor-critic learning, yet their efficacy depends critically on how they are aggregated. Current methods typically rely on static rules or task-specific hyperparameters to balance o…
Reinforcement LearningContinuous Control