Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learning

2024-02-13 · Eslam Eldeeb, Houssem Sifaou, Osvaldo Simeone, Mohammad Shehab, Hirley Alves

Reinforcement learning (RL) has been widely adopted for controlling and optimizing complex engineering systems such as next-generation wireless networks. An important challenge in adopting RL is the need for direct access to the physical environment. This limitation is particularly severe in multi-agent systems, for which conventional multi-agent reinforcement learning (MARL) requires a large number of coordinated online interactions with the environment during training. When only offline data is available, a direct application of online MARL schemes would generally fail due to the epistemic uncertainty entailed by the lack of exploration during training. In this work, we propose an offline MARL scheme that integrates distributional RL and conservative Q-learning to address the environment's inherent aleatoric uncertainty and the epistemic uncertainty arising from the use of offline data. We explore both independent and joint learning strategies. The proposed MARL scheme, referred to as multi-agent conservative quantile regression, addresses general risk-sensitive design criteria and is applied to the trajectory planning problem in drone networks, showcasing its advantages.

📄 PDF Abstract BibTeX arXiv:2402.08421

Code (1)

eslam211/conservative-and-distributional-marl 공식 구현 pytorch

Tasks

Multi-agent Reinforcement LearningQ-Learningquantile regressionreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Trajectory Planning

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Conservative Offline Policy Adaptation in Multi-Agent Games

2023-09-21 · NeurIPS 2023 11

Prior research on policy adaptation in multi-agent games has often relied on online interaction with the target agent in training, which can be expensive and impractical in real-world scenarios. Inspired by recent progre…

Conservative Offline Distributional Reinforcement Learning

2021-07-12 · NeurIPS 2021 12 · Yecheng Jason Ma, Dinesh Jayaraman, Osbert Bastani

Many reinforcement learning (RL) problems in practice are offline, learning purely from observational data. A key challenge is how to ensure the learned policy is safe, which requires quantifying the risk associated with…

D4RL Distributional Reinforcement Learning MuJoCo Offline RL+4

Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning

2023-09-22 · NeurIPS 2023 11 · Jianzhun Shao, Yun Qu, Chen Chen, Hongchang Zhang 외

Offline multi-agent reinforcement learning is challenging due to the coupling effect of both distribution shift issue common in offline setting and the high dimension issue common in multi-agent setting, making the actio…

counterfactual Multi-agent Reinforcement Learning Offline RL Q-Learning+2

Few is More: Task-Efficient Skill-Discovery for Multi-Task Offline Multi-Agent Reinforcement Learning

2025-02-13 · Xun Wang, Zhuoran Li, Hai Zhong, Longbo Huang

As a data-driven approach, offline MARL learns superior policies solely from offline datasets, ideal for domains rich in historical data but with high interaction costs and risks. However, most existing methods are task-…

Learning to Execute Multi-agent Reinforcement Learning Q-Learning

Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification

2021-11-22 · Ling Pan, Longbo Huang, Tengyu Ma, Huazhe Xu

Conservatism has led to significant progress in offline reinforcement learning (RL) where an agent learns from pre-collected datasets. However, as many real-world scenarios involve interaction among multiple agents, it i…

Continuous Control Multi-agent Reinforcement Learning Offline RL reinforcement-learning+1