Distributional Reinforcement Learning via the Cramér Distance

2026-04-26 · Vanya Aziz, Ivo Nowak, E. M. T Hendrix arxiv

This paper explores the application of the Soft Actor-Critic (SAC) algorithm within a Distributional Reinforcement Learning setting and introduces an implementation of such algorithm named Cramér-based Distributional Soft Actor-Critic (C-DSAC). The novel approach employs distributional reinforcement learning to represent state-action values, and minimizes the squared Cramér distance for learning the distribution. Empirical results across various robotic benchmarks indicate that our algorithm surpasses the performance of baseline SAC and contemporary distributional methods, with the performance advantage becoming increasingly pronounced in high-complexity environments. To explain the efficiency of the new approach, we conduct an analysis showing that its superior performance is partly due to \textit{confidence-driven} Q-value updates: High-variance target distributions (low confidence in target) lead to more conservative model updates, thereby attenuating the impact of overestimated values. This work deepens the understanding of distributional reinforcement learning, offering insights into the algorithmic mechanisms governing convergence and value estimation.

📄 PDF Abstract BibTeX arXiv:2605.08104

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

Distributional reinforcement learning with linear function approximation

2019-02-08 · Marc G. Bellemare, Nicolas Le Roux, Pablo Samuel Castro, Subhodeep Moitra

Despite many algorithmic advances, our theoretical understanding of practical distributional reinforcement learning methods remains limited. One exception is Rowland et al. (2018)'s analysis of the C51 algorithm in terms…

Distributional Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A Cramér Distance perspective on Quantile Regression based Distributional Reinforcement Learning

2021-10-01 · NeurIPS 2021 12 · Alix Lhéritier, Nicolas Bondoux

Distributional reinforcement learning (DRL) extends the value-based approach by approximating the full distribution over future returns instead of the mean only, providing a richer signal that leads to improved performan…

Distributional Reinforcement Learning quantile regression regression reinforcement-learning+1

Balanced Marginal and Joint Distributional Learning via Mixture Cramer-Wold Distance

2023-12-06 · SeungHwan An, Sungchul Hong, Jong-June Jeon

In the process of training a generative model, it becomes essential to measure the discrepancy between two high-dimensional probability distributions: the generative distribution and the ground-truth distribution of the …

Synthetic Data Generation

An Analysis of Categorical Distributional Reinforcement Learning

2018-02-22 · Mark Rowland, Marc G. Bellemare, Will Dabney, Rémi Munos 외

Distributional approaches to value-based reinforcement learning model the entire distribution of returns, rather than just their expected values, and have recently been shown to yield state-of-the-art empirical performan…

Distributional Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Cramer Type Distances for Learning Gaussian Mixture Models by Gradient Descent

2023-07-13 · Ruichong Zhang

The learning of Gaussian Mixture Models (also referred to simply as GMMs) plays an important role in machine learning. Known for their expressiveness and interpretability, Gaussian mixture models have a wide range of app…

Distributional Reinforcement Learning