Wasserstein Actor-Critic: Directed Exploration via Optimism for Continuous-Actions Control

2023-03-04 · Amarildo Likmeta, Matteo Sacco, Alberto Maria Metelli, Marcello Restelli

Uncertainty quantification has been extensively used as a means to achieve efficient directed exploration in Reinforcement Learning (RL). However, state-of-the-art methods for continuous actions still suffer from high sample complexity requirements. Indeed, they either completely lack strategies for propagating the epistemic uncertainty throughout the updates, or they mix it with aleatoric uncertainty while learning the full return distribution (e.g., distributional RL). In this paper, we propose Wasserstein Actor-Critic (WAC), an actor-critic architecture inspired by the recent Wasserstein Q-Learning (WQL) \citep{wql}, that employs approximate Q-posteriors to represent the epistemic uncertainty and Wasserstein barycenters for uncertainty propagation across the state-action space. WAC enforces exploration in a principled way by guiding the policy learning process with the optimization of an upper bound of the Q-value estimates. Furthermore, we study some peculiar issues that arise when using function approximation, coupled with the uncertainty estimation, and propose a regularized loss for the uncertainty estimation. Finally, we evaluate our algorithm on standard MujoCo tasks as well as suite of continuous-actions domains, where exploration is crucial, in comparison with state-of-the-art baselines.

📄 PDF Abstract BibTeX arXiv:2303.02378

Code (0)

등록된 구현이 없습니다.

Tasks

MuJoCoQ-LearningReinforcement Learning (RL)Uncertainty Quantification

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Wasserstein Barycenter Soft Actor-Critic

2025-06-11 · Zahra Shahrooei, Ali Baheri

Deep off-policy actor-critic algorithms have emerged as the leading framework for reinforcement learning in continuous control domains. However, most of these algorithms suffer from poor sample efficiency, especially in …

continuous-control Continuous Control MuJoCo

Exploring Pessimism and Optimism Dynamics in Deep Reinforcement Learning

2024-06-06 · Bahareh Tasdighi, Nicklas Werge, Yi-Shan Wu, Melih Kandemir

Off-policy actor-critic algorithms have shown promise in deep reinforcement learning for continuous control tasks. Their success largely stems from leveraging pessimistic state-action value function updates, which effect…

continuous-control Continuous Control Deep Reinforcement Learning reinforcement-learning+1

Better Exploration with Optimistic Actor-Critic

2019-10-28 · Kamil Ciosek, Quan Vuong, Robert Loftin, Katja Hofmann

Actor-critic methods, a type of model-free Reinforcement Learning, have been successfully applied to challenging tasks in continuous control, often achieving state-of-the art performance. However, wide-scale adoption of …

continuous-control Continuous Control Efficient Exploration Reinforcement Learning

Better Exploration with Optimistic Actor Critic

2019-12-01 · NeurIPS 2019 12 · Kamil Ciosek, Quan Vuong, Robert Loftin, Katja Hofmann

continuous-control Continuous Control Efficient Exploration Reinforcement Learning

Context-Dependent Upper-Confidence Bounds for Directed Exploration

2018-11-15 · NeurIPS 2018 12 · Raksha Kumaraswamy, Matthew Schlegel, Adam White, Martha White

Directed exploration strategies for reinforcement learning are critical for learning an optimal policy in a minimal number of interactions with the environment. Many algorithms use optimism to direct exploration, either …

Efficient Exploration Reinforcement Learning