MESA: Cooperative Meta-Exploration in Multi-Agent Learning through Exploiting State-Action Space Structure

2024-05-01 · Zhicheng Zhang, Yancheng Liang, Yi Wu, Fei Fang

Multi-agent reinforcement learning (MARL) algorithms often struggle to find strategies close to Pareto optimal Nash Equilibrium, owing largely to the lack of efficient exploration. The problem is exacerbated in sparse-reward settings, caused by the larger variance exhibited in policy learning. This paper introduces MESA, a novel meta-exploration method for cooperative multi-agent learning. It learns to explore by first identifying the agents' high-rewarding joint state-action subspace from training tasks and then learning a set of diverse exploration policies to "cover" the subspace. These trained exploration policies can be integrated with any off-policy MARL algorithm for test-time tasks. We first showcase MESA's advantage in a multi-step matrix game. Furthermore, experiments show that with learned exploration policies, MESA achieves significantly better performance in sparse-reward tasks in several multi-agent particle environments and multi-agent MuJoCo environments, and exhibits the ability to generalize to more challenging tasks at test time.

📄 PDF Abstract BibTeX arXiv:2405.00902

Code (0)

등록된 구현이 없습니다.

Tasks

Efficient ExplorationMuJoCoMulti-agent Reinforcement Learning

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

MESA: Offline Meta-RL for Safe Adaptation and Fault Tolerance

2021-12-07 · Michael Luo, Ashwin Balakrishna, Brijen Thananjeyan, Suraj Nair 외

Safe exploration is critical for using reinforcement learning (RL) in risk-sensitive environments. Recent work learns risk measures which measure the probability of violating constraints, which can then be used to enable…

continuous-control Continuous Control Meta-Learning Reinforcement Learning (RL)+1

TimeSage-MT: A Multi-Turn Benchmark for Evaluating Agentic Time Series Reasoning

2026-05-31 · Yaxuan Kong, Qingren Yao, Yuqi Nie, Yichen Li 외 arxiv

Time series data inform critical decisions across many real-world domains. While large language model (LLM) agents can analyze data through natural language and tools, it remains unclear whether they can conduct reliable…

Time Series Analysis Anomaly Detection Decision Making

MESA: Boost Ensemble Imbalanced Learning with MEta-SAmpler

2020-10-17 · NeurIPS 2020 12 · Zhining Liu, Pengfei Wei, Jing Jiang, Wei Cao 외

Imbalanced learning (IL), i.e., learning unbiased models from class-imbalanced data, is a challenging problem. Typical IL methods including resampling and reweighting were designed based on some heuristic assumptions. Th…

imbalanced classification Meta-Learning

Conditionally Optimistic Exploration for Cooperative Deep Multi-Agent Reinforcement Learning

2023-03-16 · Xutong Zhao, Yangchen Pan, Chenjun Xiao, Sarath Chandar 외

Efficient exploration is critical in cooperative deep Multi-Agent Reinforcement Learning (MARL). In this work, we propose an exploration method that effectively encourages cooperative exploration based on the idea of seq…

Efficient Exploration Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Emergence of Collective Open-Ended Exploration from Decentralized Meta-Reinforcement Learning

2023-11-01 · Richard Bornemann, Gautier Hamon, Eleni Nisioti, Clément Moulin-Frier

Recent works have proven that intricate cooperative behaviors can emerge in agents trained using meta reinforcement learning on open ended task distributions using self-play. While the results are impressive, we argue th…

Meta Reinforcement Learning reinforcement-learning