Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms
Multi-Agent Reinforcement Learning (MARL) has shown promising results across several domains. Despite this promise, MARL policies often lack robustness and are therefore sensitive to small changes in their environment. This presents a serious concern for the real world deployment of MARL algorithms, where the testing environment may slightly differ from the training environment. In this work we show that we can gain robustness by controlling a policy’s Lipschitz constant, and under mild conditions, establish the existence of a Lipschitz and close-to-optimal policy. Motivated by these insights, we propose a new robust MARL framework, ERNIE, that promotes the Lipschitz continuity of the policies with respect to the state observations and actions by adversarial regularization. The ERNIE framework provides robustness against noisy observations, changing transition dynamics, and malicious actions of agents. However, ERNIE’s adversarial regularization may introduce some training instability. To reduce this instability, we reformulate adversarial regularization as a Stackelberg game. We demonstrate the effectiveness of the proposed framework with extensive experiments in traffic light control and particle environments. In addition, we extend ERNIE to mean-field MARL with a formulation based on distributionally robust optimization that outperforms its non-robust counterpart and is of independent interest. Our code is available at https://github.com/abukharin3/ERNIE.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations
A deep reinforcement learning (DRL) agent observes its states through observations, which may contain natural measurement errors or adversarial noises. Since the observations deviate from the true states, they can mislea…
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Sparse Adversarial Attack in Multi-agent Reinforcement Learning
Cooperative multi-agent reinforcement learning (cMARL) has many real applications, but the policy trained by existing cMARL algorithms is not robust enough when deployed. There exist also many methods about adversarial a…
Adversarial AttackMulti-agent Reinforcement Learningreinforcement-learningReinforcement Learning+1Conditioning of Reinforcement Learning Agents and its Policy Regularization Application
The outcome of Jacobian singular values regularization was studied for supervised learning problems. It also was shown that Jacobian conditioning regularization can help to avoid the ``mode-collapse'' problem in Generati…
continuous-controlContinuous Controlreinforcement-learningReinforcement Learning+1The Benefits of Power Regularization in Cooperative Reinforcement Learning
Cooperative Multi-Agent Reinforcement Learning (MARL) algorithms, trained only to optimize task reward, can lead to a concentration of power where the failure or adversarial intent of a single agent could decimate the re…
Multi-agent Reinforcement Learningreinforcement-learningReinforcement LearningRoMFAC: A robust mean-field actor-critic reinforcement learning against adversarial perturbations on states
Multi-agent deep reinforcement learning makes optimal decisions dependent on system states observed by agents, but any uncertainty on the observations may mislead agents to take wrong actions. The Mean-Field Actor-Critic…
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)