CTD4 -- A Deep Continuous Distributional Actor-Critic Agent with a Kalman Fusion of Multiple Critics

2024-05-04 · David Valencia, Henry Williams, Trevor Gee, Bruce A MacDonald, Minas Liarokapis

Categorical Distributional Reinforcement Learning (CDRL) has demonstrated superior sample efficiency in learning complex tasks compared to conventional Reinforcement Learning (RL) approaches. However, the practical application of CDRL is encumbered by challenging projection steps, detailed parameter tuning, and domain knowledge. This paper addresses these challenges by introducing a pioneering Continuous Distributional Model-Free RL algorithm tailored for continuous action spaces. The proposed algorithm simplifies the implementation of distributional RL, adopting an actor-critic architecture wherein the critic outputs a continuous probability distribution. Additionally, we propose an ensemble of multiple critics fused through a Kalman fusion mechanism to mitigate overestimation bias. Through a series of experiments, we validate that our proposed method is easy to train and serves as a sample-efficient solution for executing complex continuous-control tasks.

📄 PDF Abstract BibTeX arXiv:2405.02576

Code (1)

UoA-CARES/cares_reinforcement_learning/blob/main/cares_reinforcement_learning/algorithm/policy/CTD4.py 공식 구현 pytorch

Tasks

continuous-control Continuous ControlDistributional Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Continuous Control Reinforcement Learning: Distributed Distributional DrQ Algorithms

2024-04-16 · Zehao Zhou

Distributed Distributional DrQ is a model-free and off-policy RL algorithm for continuous control tasks based on the state and observation of the agent, which is an actor-critic method with the data-augmentation and the …

continuous-control Continuous Control Data Augmentation reinforcement-learning+1

Distributional Policy Optimization: An Alternative Approach for Continuous Control

2019-05-23 · NeurIPS 2019 12 · Chen Tessler, Guy Tennenholtz, Shie Mannor

We identify a fundamental problem in policy gradient-based methods in continuous control. As policy gradient methods require the agent's underlying probability distribution, they limit policy representation to parametric…

continuous-control Continuous Control Policy Gradient Methods

DSAC: Distributional Soft Actor Critic for Risk-Sensitive Reinforcement Learning

2020-04-30 · Xiaoteng Ma, Li Xia, Zhengyuan Zhou, Jun Yang 외

In this paper, we present a new reinforcement learning (RL) algorithm called Distributional Soft Actor Critic (DSAC), which exploits the distributional information of accumulated rewards to achieve better performance. Se…

continuous-control Continuous Control reinforcement-learning Reinforcement Learning+1

On the Steady-State Distributionally Robust Kalman Filter

2025-03-31 · MinHyuk Jang, Astghik Hakobyan, Insoon Yang

State estimation in the presence of uncertain or data-driven noise distributions remains a critical challenge in control and robotics. Although the Kalman filter is the most popular choice, its performance degrades signi…

Computational Efficiency State Estimation

DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning

2021-02-16 · Wei-Fang Sun, Cheng-Kuang Lee, Chun-Yi Lee

In fully cooperative multi-agent reinforcement learning (MARL) settings, the environments are highly stochastic due to the partial observability of each agent and the continuously changing policies of the other agents. T…

Multi-agent Reinforcement Learning Q-Learning SMAC SMAC++1