Offline Decentralized Multi-Agent Reinforcement Learning

2021-08-04 · Jiechuan Jiang, Zongqing Lu

In many real-world multi-agent cooperative tasks, due to high cost and risk, agents cannot continuously interact with the environment and collect experiences during learning, but have to learn from offline datasets. However, the transition dynamics in the dataset of each agent can be much different from the ones induced by the learned policies of other agents in execution, creating large errors in value estimates. Consequently, agents learn uncoordinated low-performing policies. In this paper, we propose a framework for offline decentralized multi-agent reinforcement learning, which exploits value deviation and transition normalization to deliberately modify the transition probabilities. Value deviation optimistically increases the transition probabilities of high-value next states, and transition normalization normalizes the transition probabilities of next states. They together enable agents to learn high-performing and coordinated policies. Theoretically, we prove the convergence of Q-learning under the altered non-stationary transition dynamics. Empirically, we show that the framework can be easily built on many existing offline reinforcement learning algorithms and achieve substantial improvement in a variety of multi-agent tasks.

📄 PDF Abstract BibTeX arXiv:2108.01832

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement LearningQ-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Online Tuning for Offline Decentralized Multi-Agent Reinforcement Learning

2021-09-29 · Jiechuan Jiang, Zongqing Lu

Offline reinforcement learning could learn effective policies from a fixed dataset, which is promising in real-world applications. However, in offline decentralized multi-agent reinforcement learning, due to the discrepa…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Offline-to-Online Multi-Agent Reinforcement Learning with Offline Value Function Memory and Sequential Exploration

2024-10-25 · Hai Zhong, Xun Wang, Zhuoran Li, Longbo Huang

Offline-to-Online Reinforcement Learning has emerged as a powerful paradigm, leveraging offline data for initialization and online fine-tuning to enhance both sample efficiency and performance. However, most existing res…

Efficient Exploration Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning+3

Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learning

2021-02-08 · Xueguang Lyu, Yuchen Xiao, Brett Daley, Christopher Amato

Centralized Training for Decentralized Execution, where agents are trained offline using centralized information but execute in a decentralized manner online, has gained popularity in the multi-agent reinforcement learni…

Misconceptions Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning+1

On Centralized Critics in Multi-Agent Reinforcement Learning

2024-08-26 · Xueguang Lyu, Andrea Baisero, Yuchen Xiao, Brett Daley 외

Centralized Training for Decentralized Execution where agents are trained offline in a centralized fashion and execute online in a decentralized manner, has become a popular approach in Multi-Agent Reinforcement Learning…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Representation Learning

MADiff: Offline Multi-agent Learning with Diffusion Models

2023-05-27 · Zhengbang Zhu, Minghuan Liu, Liyuan Mao, Bingyi Kang 외

Offline reinforcement learning (RL) aims to learn policies from pre-existing datasets without further interactions, making it a challenging task. Q-learning algorithms struggle with extrapolation errors in offline settin…

Offline RL Q-Learning Reinforcement Learning (RL)Trajectory Prediction