Scalable Multi-agent Covering Option Discovery based on Kronecker Graphs

2023-07-21 · Jiayu Chen, Jingdi Chen, Tian Lan, Vaneet Aggarwal

Covering skill (a.k.a., option) discovery has been developed to improve the exploration of RL in single-agent scenarios with sparse reward signals, through connecting the most distant states in the embedding space provided by the Fiedler vector of the state transition graph. Given that joint state space grows exponentially with the number of agents in multi-agent systems, existing researches still relying on single-agent skill discovery either become prohibitive or fail to directly discover joint skills that improve the connectivity of the joint state space. In this paper, we propose multi-agent skill discovery which enables the ease of decomposition. Our key idea is to approximate the joint state space as a Kronecker graph, based on which we can directly estimate its Fiedler vector using the Laplacian spectrum of individual agents' transition graphs. Further, considering that directly computing the Laplacian spectrum is intractable for tasks with infinite-scale state spaces, we further propose a deep learning extension of our method by estimating eigenfunctions through NN-based representation learning techniques. The evaluation on multi-agent tasks built with simulators like Mujoco, shows that the proposed algorithm can successfully identify multi-agent skills, and significantly outperforms the state-of-the-art. Codes are available at: https://github.itap.purdue.edu/Clan-labs/Scalable_MAOD_via_KP.

📄 PDF Abstract BibTeX arXiv:2307.11629

Code (0)

등록된 구현이 없습니다.

Tasks

MuJoCoRepresentation Learning

Methods 이 논문이 사용한 방법론

fail 설명 없음

Multi-agent Deep Covering Skill Discovery

2022-10-07 · Jiayu Chen, Marina Haliem, Tian Lan, Vaneet Aggarwal

The use of skills (a.k.a., options) can greatly accelerate exploration in reinforcement learning, especially when only sparse reward signals are available. While option discovery methods have been proposed for individual…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)

Learning Multi-agent Skills for Tabular Reinforcement Learning using Factor Graphs

2022-01-20 · Jiayu Chen, Jingdi Chen, Tian Lan, Vaneet Aggarwal

Covering skill (a.k.a., option) discovery has been developed to improve the exploration of reinforcement learning in single-agent scenarios with sparse reward signals, through connecting the most distant states in the em…

reinforcement-learning Reinforcement Learning (RL)

Discovery of Options via Meta-Learned Subgoals

2021-02-12 · NeurIPS 2021 12 · Vivek Veeriah, Tom Zahavy, Matteo Hessel, Zhongwen Xu 외

Temporal abstractions in the form of options have been shown to help reinforcement learning (RL) agents learn faster. However, despite prior work on this topic, the problem of discovering options through interaction with…

Reinforcement Learning (RL)

Option Discovery using Deep Skill Chaining

2020-05-01 · ICLR 2020 1 · Akhil Bagaria, George Konidaris

Autonomously discovering temporally extended actions, or skills, is a longstanding goal of hierarchical reinforcement learning. We propose a new algorithm that combines skill chaining with deep neural networks to autonom…

continuous-control Continuous Control Hierarchical Reinforcement Learning reinforcement-learning+2

Inter-Agent Relative Representations for Multi-Agent Option Discovery

2025-12-31 · Raul D. Steleac, Mohan Sridharan, David Abel arxiv

Temporally extended actions improve the ability to explore and plan in single-agent settings. In multi-agent settings, the exponential growth of the joint state space with the number of agents makes coordinated behaviour…