Robust Multi-Agent Reinforcement Learning with Model Uncertainty
In this work, we study the problem of multi-agent reinforcement learning (MARL) with model uncertainty, which is referred to as robust MARL. This is naturally motivated by some multi-agent applications where each agent may not have perfectly accurate knowledge of the model, e.g., all the reward functions of other agents. Little a priori work on MARL has accounted for such uncertainties, neither in problem formulation nor in algorithm design. In contrast, we model the problem as a robust Markov game, where the goal of all agents is to find policies such that no agent has the incentive to deviate, i.e., reach some equilibrium point, which is also robust to the possible uncertainty of the MARL model. We first introduce the solution concept of robust Nash equilibrium in our setting, and develop a Q-learning algorithm to find such equilibrium policies, with convergence guarantees under certain conditions. In order to handle possibly enormous state-action spaces in practice, we then derive the policy gradients for robust MARL, and develop an actor-critic algorithm with function approximation. Our experiments demonstrate that the proposed algorithm outperforms several baseline MARL methods that do not account for the model uncertainty, in several standard but uncertain cooperative and competitive MARL environments.
Code (0)
등록된 구현이 없습니다.
Tasks
modelMulti-agent Reinforcement LearningQ-Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Review of Uncertainty for Deep Reinforcement Learning
Uncertainty is ubiquitous in games, both in the agents playing games and often in the games themselves. Working with uncertainty is therefore an important component of successful deep reinforcement learning agents. While…
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Exploring Agentic Tool-Calling Decisions via Uncertainty-Aligned Reinforcement Learning
Large language model (LLM)-based agents often make suboptimal tool-use decisions, including unsupported tool invocation and hallucinated direct responses, which may accumulate errors throughout multi-step interactions. E…
Reinforcement LearningRisk-Sensitive Bayesian Games for Multi-Agent Reinforcement Learning under Policy Uncertainty
In stochastic games with incomplete information, the uncertainty is evoked by the lack of knowledge about a player's own and the other players' types, i.e. the utility function and the policy space, and also the inherent…
Multi-agent Reinforcement Learningreinforcement-learningReinforcement Learning (RL)Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learning
Reinforcement learning (RL) has been widely adopted for controlling and optimizing complex engineering systems such as next-generation wireless networks. An important challenge in adopting RL is the need for direct acces…
Multi-agent Reinforcement LearningQ-Learningquantile regressionreinforcement-learning+3Safe multi-agent motion planning under uncertainty for drones using filtered reinforcement learning
We consider the problem of safe multi-agent motion planning for drones in uncertain, cluttered workspaces. For this problem, we present a tractable motion planner that builds upon the strengths of reinforcement learning …
Collision AvoidanceMotion Planningreinforcement-learningReinforcement Learning+1