Influence-Based Multi-Agent Exploration

2019-10-12 · ICLR 2020 1 · Tonghan Wang, Jianhao Wang, Yi Wu, Chongjie Zhang

Intrinsically motivated reinforcement learning aims to address the exploration challenge for sparse-reward tasks. However, the study of exploration methods in transition-dependent multi-agent settings is largely absent from the literature. We aim to take a step towards solving this problem. We present two exploration methods: exploration via information-theoretic influence (EITI) and exploration via decision-theoretic influence (EDTI), by exploiting the role of interaction in coordinated behaviors of agents. EITI uses mutual information to capture influence transition dynamics. EDTI uses a novel intrinsic reward, called Value of Interaction (VoI), to characterize and quantify the influence of one agent's behavior on expected returns of other agents. By optimizing EITI or EDTI objective as a regularizer, agents are encouraged to coordinate their exploration and learn policies to optimize team performance. We show how to optimize these regularizers so that they can be easily integrated with policy gradient reinforcement learning. The resulting update rule draws a connection between coordinated exploration and intrinsic reward distribution. Finally, we empirically demonstrate the significant strength of our method in a variety of multi-agent scenarios.

📄 PDF Abstract BibTeX arXiv:1910.05512

Code (1)

TonghanWang/EITI-EDTI 공식 구현 tf

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

PIMAEX: Multi-Agent Exploration through Peer Incentivization

2025-01-02 · Michael Kölle, Johannes Tochtermann, Julian Schönberger, Gerhard Stenzel 외

While exploration in single-agent reinforcement learning has been studied extensively in recent years, considerably less work has focused on its counterpart in multi-agent reinforcement learning. To address this issue, t…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning

Settling Decentralized Multi-Agent Coordinated Exploration by Novelty Sharing

2024-02-03 · Haobin Jiang, Ziluo Ding, Zongqing Lu

Exploration in decentralized cooperative multi-agent reinforcement learning faces two challenges. One is that the novelty of global states is unavailable, while the novelty of local observations is biased. The other is h…

Multi-agent Reinforcement Learning

Credit Assignment and Efficient Exploration based on Influence Scope in Multi-agent Reinforcement Learning

2025-05-13 · Shuai Han, Mehdi Dastani, Shihan Wang

Training cooperative agents in sparse-reward scenarios poses significant challenges for multi-agent reinforcement learning (MARL). Without clear feedback on actions at each step in sparse-reward setting, previous methods…

Efficient Exploration Multi-agent Reinforcement Learning

Situation-Dependent Causal Influence-Based Cooperative Multi-agent Reinforcement Learning

2023-12-15 · Xiao Du, Yutong Ye, Pengyu Zhang, Yaning Yang 외

Learning to collaborate has witnessed significant progress in multi-agent reinforcement learning (MARL). However, promoting coordination among agents and enhancing exploration capabilities remain challenges. In multi-age…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning

EXPODE: EXploiting POlicy Discrepancy for Efficient Exploration in Multi-agent Reinforcement Learning

2023-05-30 · International Conference on Autonomous Agents and Multiagent Systems 2023 5 · Yucong Zhang, Chao Yu

Recently, Multi-Agent Reinforcement Learning (MARL) has been applied to a large number of scenarios and has shown promising performance. However, existing MARL algorithms still suffer from the severe exploration problem.…

Efficient Exploration Multi-agent Reinforcement Learning Starcraft Starcraft II