Robust Multi-Agent Reinforcement Learning with State Uncertainty

2023-07-30 · Sihong He, Songyang Han, Sanbao Su, Shuo Han, Shaofeng Zou, Fei Miao

In real-world multi-agent reinforcement learning (MARL) applications, agents may not have perfect state information (e.g., due to inaccurate measurement or malicious attacks), which challenges the robustness of agents' policies. Though robustness is getting important in MARL deployment, little prior work has studied state uncertainties in MARL, neither in problem formulation nor algorithm design. Motivated by this robustness issue and the lack of corresponding studies, we study the problem of MARL with state uncertainty in this work. We provide the first attempt to the theoretical and empirical analysis of this challenging problem. We first model the problem as a Markov Game with state perturbation adversaries (MG-SPA) by introducing a set of state perturbation adversaries into a Markov Game. We then introduce robust equilibrium (RE) as the solution concept of an MG-SPA. We conduct a fundamental analysis regarding MG-SPA such as giving conditions under which such a robust equilibrium exists. Then we propose a robust multi-agent Q-learning (RMAQ) algorithm to find such an equilibrium, with convergence guarantees. To handle high-dimensional state-action space, we design a robust multi-agent actor-critic (RMAAC) algorithm based on an analytical expression of the policy gradient derived in the paper. Our experiments show that the proposed RMAQ algorithm converges to the optimal value function; our RMAAC algorithm outperforms several MARL and robust MARL methods in multiple multi-agent environments when state uncertainty is present. The source code is public on \url{https://github.com/sihongho/robust_marl_with_state_uncertainty}.

📄 PDF Abstract BibTeX arXiv:2307.16212

Code (1)

sihongho/robust_marl_with_state_uncertainty 공식 구현 tf

Tasks

Multi-agent Reinforcement LearningQ-Learningreinforcement-learning Reinforcement LearningSingle Particle Analysis

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Risk Sensitive Model-Based Reinforcement Learning using Uncertainty Guided Planning

2021-11-09 · Stefan Radic Webster, Peter Flach

Identifying uncertainty and taking mitigating actions is crucial for safe and trustworthy reinforcement learning agents, especially when deployed in high-risk environments. In this paper, risk sensitivity is promoted in …

Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Toward Risk-based Optimistic Exploration for Cooperative Multi-Agent Reinforcement Learning

2023-03-03 · Jihwan Oh, Joonkee Kim, Minchan Jeong, Se-Young Yun

The multi-agent setting is intricate and unpredictable since the behaviors of multiple agents influence one another. To address this environmental uncertainty, distributional reinforcement learning algorithms that incorp…

Distributional Reinforcement Learning Multi-agent Reinforcement Learning quantile regression reinforcement-learning+2

Safe multi-agent motion planning under uncertainty for drones using filtered reinforcement learning

2023-10-31 · Sleiman Safaoui, Abraham P. Vinod, Ankush Chakrabarty, Rien Quirynen 외

We consider the problem of safe multi-agent motion planning for drones in uncertain, cluttered workspaces. For this problem, we present a tractable motion planner that builds upon the strengths of reinforcement learning …

Collision Avoidance Motion Planning reinforcement-learning Reinforcement Learning+1

Query The Agent: Improving sample efficiency through epistemic uncertainty estimation

2022-10-05 · Julian Alverio, Boris Katz, Andrei Barbu

Curricula for goal-conditioned reinforcement learning agents typically rely on poor estimates of the agent's epistemic uncertainty or fail to consider the agents' epistemic uncertainty altogether, resulting in poor sampl…

reinforcement-learning Reinforcement Learning (RL)

Decentralized Multi-Agent Reinforcement Learning for Task Offloading Under Uncertainty

2021-07-16 · Yuanchao Xu, Amal Feriani, Ekram Hossain

Multi-Agent Reinforcement Learning (MARL) is a challenging subarea of Reinforcement Learning due to the non-stationarity of the environments and the large dimensionality of the combined action space. Deep MARL algorithms…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)