EXPODE: EXploiting POlicy Discrepancy for Efficient Exploration in Multi-agent Reinforcement Learning
Recently, Multi-Agent Reinforcement Learning (MARL) has been applied to a large number of scenarios and has shown promising performance. However, existing MARL algorithms still suffer from the severe exploration problem. In this paper, we propose EXploiting POlicy Discrepancy for efficient Exploration (EXPODE), a new multi-agent exploration framework that leverages discrepancy between two different policies to enable the agents to explore the environment more efficiently. In addition, to tackle the mutual influence issue caused by the concurrent exploration of the agents, we propose three different mechanisms to coordinate the agents’ exploration by taking the information of other agents’ states and policies into account when measuring the agent-wise policy discrepancies. Experimental results on three challenging tasks, i.e., Predator Prey, StarCraft II micromanagement tasks, and Google Research Football, demonstrate that EXPODE achieves the state-of-the-art performance.
Code (1)
Tasks
Efficient ExplorationMulti-agent Reinforcement LearningStarcraftStarcraft IISimilar Papers 제목 키워드 기반
Density Estimation via Discrepancy
Given i.i.d samples from some unknown continuous density on hyper-rectangle $[0, 1]^d$, we attempt to learn a piecewise constant function that approximates this underlying density non-parametrically. Our density estimate…
Density EstimationLearning and Exploiting Multiple Subgoals for Fast Exploration in Hierarchical Reinforcement Learning
Hierarchical Reinforcement Learning (HRL) exploits temporally extended actions, or options, to make decisions from a higher-dimensional perspective to alleviate the sparse reward problem, one of the most challenging prob…
Hierarchical Reinforcement LearningMontezuma's Revengereinforcement-learningReinforcement Learning+1Reformulate LLM Reinforcement Learning for Efficient Training under Black-box Discrepancy
Reinforcement Learning (RL) has emerged as a pivotal post-training paradigm, yet it frequently suffers from unpredictable sub-optimum performance or even training collapses. Recent findings attribute these failures to a …
Reinforcement LearningSelf-Imitation Learning
This paper proposes Self-Imitation Learning (SIL), a simple off-policy actor-critic algorithm that learns to reproduce the agent's past good decisions. This algorithm is designed to verify our hypothesis that exploiting …
Atari GamesImitation LearningMuJoCoMESA: Cooperative Meta-Exploration in Multi-Agent Learning through Exploiting State-Action Space Structure
Multi-agent reinforcement learning (MARL) algorithms often struggle to find strategies close to Pareto optimal Nash Equilibrium, owing largely to the lack of efficient exploration. The problem is exacerbated in sparse-re…
Efficient ExplorationMuJoCoMulti-agent Reinforcement Learning