Distributional Reward Estimation for Effective Multi-Agent Deep Reinforcement Learning

2022-10-14 · Jifeng Hu, Yanchao Sun, Hechang Chen, Sili Huang, Haiyin Piao, Yi Chang, Lichao Sun

Multi-agent reinforcement learning has drawn increasing attention in practice, e.g., robotics and automatic driving, as it can explore optimal policies using samples generated by interacting with the environment. However, high reward uncertainty still remains a problem when we want to train a satisfactory model, because obtaining high-quality reward feedback is usually expensive and even infeasible. To handle this issue, previous methods mainly focus on passive reward correction. At the same time, recent active reward estimation methods have proven to be a recipe for reducing the effect of reward uncertainty. In this paper, we propose a novel Distributional Reward Estimation framework for effective Multi-Agent Reinforcement Learning (DRE-MARL). Our main idea is to design the multi-action-branch reward estimation and policy-weighted reward aggregation for stabilized training. Specifically, we design the multi-action-branch reward estimation to model reward distributions on all action branches. Then we utilize reward aggregation to obtain stable updating signals during training. Our intuition is that consideration of all possible consequences of actions could be useful for learning policies. The superiority of the DRE-MARL is demonstrated using benchmark multi-agent scenarios, compared with the SOTA baselines in terms of both effectiveness and robustness.

📄 PDF Abstract BibTeX arXiv:2210.07636

Code (1)

jf-hu/dre-marl 공식 구현

Tasks

Deep Reinforcement LearningMulti-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Noise Distribution Decomposition based Multi-Agent Distributional Reinforcement Learning

2023-12-12 · Wei Geng, Baidi Xiao, Rongpeng Li, Ning Wei 외

Generally, Reinforcement Learning (RL) agent updates its policy by repetitively interacting with the environment, contingent on the received rewards to observed states and undertaken actions. However, the environmental d…

Distributional Reinforcement Learning Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning+1

2048: Reinforcement Learning in a Delayed Reward Environment

2025-07-07 · Prady Saligram, Tanvir Bhathal, Robby Manihani

Delayed and sparse rewards present a fundamental obstacle for reinforcement-learning (RL) agents, which struggle to assign credit for actions whose benefits emerge many steps later. The sliding-tile game 2048 epitomizes …

quantile regression reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Distributional Reward Decomposition for Reinforcement Learning

2019-11-06 · NeurIPS 2019 12 · Zichuan Lin, Li Zhao, Derek Yang, Tao Qin 외

Many reinforcement learning (RL) tasks have specific properties that can be leveraged to modify existing RL algorithms to adapt to those tasks and further improve performance, and a general class of such properties is th…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

DSAC: Distributional Soft Actor Critic for Risk-Sensitive Reinforcement Learning

2020-04-30 · Xiaoteng Ma, Li Xia, Zhengyuan Zhou, Jun Yang 외

In this paper, we present a new reinforcement learning (RL) algorithm called Distributional Soft Actor Critic (DSAC), which exploits the distributional information of accumulated rewards to achieve better performance. Se…

continuous-control Continuous Control reinforcement-learning Reinforcement Learning+1

Distributional Advantage Actor-Critic

2018-06-10 · Li Shangda, Bing Selina, Yang Steven

In traditional reinforcement learning, an agent maximizes the reward collected during its interaction with the environment by approximating the optimal policy through the estimation of value functions. Typically, given a…

Q-Learning quantile regression