Noise Distribution Decomposition based Multi-Agent Distributional Reinforcement Learning
Generally, Reinforcement Learning (RL) agent updates its policy by repetitively interacting with the environment, contingent on the received rewards to observed states and undertaken actions. However, the environmental disturbance, commonly leading to noisy observations (e.g., rewards and states), could significantly shape the performance of agent. Furthermore, the learning performance of Multi-Agent Reinforcement Learning (MARL) is more susceptible to noise due to the interference among intelligent agents. Therefore, it becomes imperative to revolutionize the design of MARL, so as to capably ameliorate the annoying impact of noisy rewards. In this paper, we propose a novel decomposition-based multi-agent distributional RL method by approximating the globally shared noisy reward by a Gaussian mixture model (GMM) and decomposing it into the combination of individual distributional local rewards, with which each agent can be updated locally through distributional RL. Moreover, a diffusion model (DM) is leveraged for reward generation in order to mitigate the issue of costly interaction expenditure for learning distributions. Furthermore, the optimality of the distribution decomposition is theoretically validated, while the design of loss function is carefully calibrated to avoid the decomposition ambiguity. We also verify the effectiveness of the proposed method through extensive simulation experiments with noisy rewards. Besides, different risk-sensitive policies are evaluated in order to demonstrate the superiority of distributional RL in different MARL tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Distributional Reinforcement LearningMulti-agent Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Distributional Reward Decomposition for Reinforcement Learning
Many reinforcement learning (RL) tasks have specific properties that can be leveraged to modify existing RL algorithms to adapt to those tasks and further improve performance, and a general class of such properties is th…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Offline Multi-agent Reinforcement Learning via Score Decomposition
Offline cooperative multi-agent reinforcement learning (MARL) faces unique challenges due to distributional shifts, particularly stemming from the high dimensionality of joint action spaces and the presence of out-of-dis…
continuous-controlContinuous ControlMuJoCoMulti-agent Reinforcement Learning+2Exploring the Robustness of Distributional Reinforcement Learning against Noisy State Observations
In real scenarios, state observations that an agent observes may contain measurement errors or adversarial noises, misleading the agent to take suboptimal actions or even collapse while training. In this paper, we study …
Distributional Reinforcement Learningreinforcement-learningReinforcement Learning (RL)Exploring the Training Robustness of Distributional Reinforcement Learning against Noisy State Observations
In real scenarios, state observations that an agent observes may contain measurement errors or adversarial noises, misleading the agent to take suboptimal actions or even collapse while training. In this paper, we study …
Density EstimationDistributional Reinforcement Learningreinforcement-learningReinforcement Learning (RL)Enabling risk-aware Reinforcement Learning for medical interventions through uncertainty decomposition
Reinforcement Learning (RL) is emerging as tool for tackling complex control and decision-making problems. However, in high-risk environments such as healthcare, manufacturing, automotive or aerospace, it is often challe…
Decision Makingreinforcement-learningReinforcement Learning (RL)