Scalable Multi-Agent Reinforcement Learning with General Utilities

2023-02-15 · Donghao Ying, Yuhao Ding, Alec Koppel, Javad Lavaei

We study the scalable multi-agent reinforcement learning (MARL) with general utilities, defined as nonlinear functions of the team's long-term state-action occupancy measure. The objective is to find a localized policy that maximizes the average of the team's local utility functions without the full observability of each agent in the team. By exploiting the spatial correlation decay property of the network structure, we propose a scalable distributed policy gradient algorithm with shadow reward and localized policy that consists of three steps: (1) shadow reward estimation, (2) truncated shadow Q-function estimation, and (3) truncated policy gradient estimation and policy update. Our algorithm converges, with high probability, to $\epsilon$-stationarity with $\widetilde{\mathcal{O}}(\epsilon^{-2})$ samples up to some approximation error that decreases exponentially in the communication radius. This is the first result in the literature on multi-agent RL with general utilities that does not require the full observability.

📄 PDF Abstract BibTeX arXiv:2302.07938

Code (0)

등록된 구현이 없습니다.

Tasks

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Scalable Primal-Dual Actor-Critic Method for Safe Multi-Agent RL with General Utilities

2023-05-27 · NeurIPS 2023 11

We investigate safe multi-agent reinforcement learning, where agents seek to collectively maximize an aggregate sum of local objectives while satisfying their own safety constraints. The objective and constraints are des…

Decision Making Multi-agent Reinforcement Learning

On Multi-Agent Inverse Reinforcement Learning

2024-11-22 · Till Freihaut, Giorgia Ramponi

In multi-agent systems, the agent behavior is highly influenced by its utility function, as these utilities shape both individual goals as well as interactions with the other agents. Inverse Reinforcement Learning (IRL) …

reinforcement-learning Reinforcement Learning

Policy Gradient for Reinforcement Learning with General Utilities

2022-10-03 · Navdeep Kumar, Kaixin Wang, Kfir Levy, Shie Mannor

In Reinforcement Learning (RL), the goal of agents is to discover an optimal policy that maximizes the expected cumulative rewards. This objective may also be viewed as finding a policy that optimizes a linear function o…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Value Under Ignorance in Universal Artificial Intelligence

2025-12-18 · Cole Wyeth, Marcus Hutter arxiv

We generalize the AIXI reinforcement learning agent to admit a wider class of utility functions. Assigning a utility to each possible interaction history forces us to confront the ambiguity that some hypotheses in the ag…

Reinforcement Learning

Solving Stochastic Games

2009-12-01 · NeurIPS 2009 12 · Liam M. Dermed, Charles L. Isbell

Solving multi-agent reinforcement learning problems has proven difficult because of the lack of tractable algorithms. We provide the first approximation algorithm which solves stochastic games to within $\epsilon$ relat…