Non-Crossing Quantile Regression for Distributional Reinforcement Learning

2020-12-01 · NeurIPS 2020 12 · Fan Zhou, Jianing Wang, Xingdong Feng

Distributional reinforcement learning (DRL) estimates the distribution over future returns instead of the mean to more efficiently capture the intrinsic uncertainty of MDPs. However, batch-based DRL algorithms cannot guarantee the non-decreasing property of learned quantile curves especially at the early training stage, leading to abnormal distribution estimates and reduced model interpretability. To address these issues, we introduce a general DRL framework by using non-crossing quantile regression to ensure the monotonicity constraint within each sampled batch, which can be incorporated with any well-known DRL algorithm. We demonstrate the validity of our method from both the theory and model implementation perspectives. Experiments on Atari 2600 Games show that some state-of-art DRL algorithms with the non-crossing modification can significantly outperform their baselines in terms of faster convergence speeds and better testing performance. In particular, our method can effectively recover the distribution information and thus dramatically increase the exploration efficiency when the reward space is extremely sparse.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Atari GamesDistributional Reinforcement Learningquantile regressionregression reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Deep Distributional Learning with Non-crossing Quantile Network

2025-04-11 · Guohao Shen, Runpeng Dai, Guojun Wu, Shikai Luo 외

In this paper, we introduce a non-crossing quantile (NQ) network for conditional distribution learning. By leveraging non-negative activation functions, the NQ network ensures that the learned distributions remain monoto…

Distributional Reinforcement Learning quantile regression Reinforcement Learning (RL)

A Cramér Distance perspective on Quantile Regression based Distributional Reinforcement Learning

2021-10-01 · NeurIPS 2021 12 · Alix Lhéritier, Nicolas Bondoux

Distributional reinforcement learning (DRL) extends the value-based approach by approximating the full distribution over future returns instead of the mean only, providing a richer signal that leads to improved performan…

Distributional Reinforcement Learning quantile regression regression reinforcement-learning+1

Distributional Reinforcement Learning with Monotonic Splines

2021-09-29 · ICLR 2022 4 · Yudong Luo, Guiliang Liu, Haonan Duan, Oliver Schulte 외

Distributional Reinforcement Learning (RL) differs from traditional RL by estimating the distribution over returns to capture the intrinsic uncertainty of MDPs. One key challenge in distributional RL lies in how to param…

Distributional Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A PyTorch Framework for Scalable Non-Crossing Quantile Regression

2025-10-25 · Kaihua Chang arxiv

Quantile regression is fundamental to distributional modeling, yet independent estimation of multiple quantiles frequently produces crossing -- where estimated quantile functions violate monotonicity, implying impossible…

Non-Crossing Deep Quantile Regression for Distributional Survival Prediction

2026-08-17 · Shuai Huang, Zhe Qu, Zhaowei Hua, Guohao Shen 외 arxiv

In survival analysis the way covariates act on the risk of an event often differs between early and late failure times, yet hazard- and mean-based summaries collapse this variation into a single number. Quantile-based mo…