Multi-Agent Advisor Q-Learning

2021-10-26 · Sriram Ganapathi Subramanian, Matthew E. Taylor, Kate Larson, Mark Crowley

In the last decade, there have been significant advances in multi-agent reinforcement learning (MARL) but there are still numerous challenges, such as high sample complexity and slow convergence to stable policies, that need to be overcome before wide-spread deployment is possible. However, many real-world environments already, in practice, deploy sub-optimal or heuristic approaches for generating policies. An interesting question that arises is how to best use such approaches as advisors to help improve reinforcement learning in multi-agent domains. In this paper, we provide a principled framework for incorporating action recommendations from online sub-optimal advisors in multi-agent settings. We describe the problem of ADvising Multiple Intelligent Reinforcement Agents (ADMIRAL) in nonrestrictive general-sum stochastic game environments and present two novel Q-learning based algorithms: ADMIRAL - Decision Making (ADMIRAL-DM) and ADMIRAL - Advisor Evaluation (ADMIRAL-AE), which allow us to improve learning by appropriately incorporating advice from an advisor (ADMIRAL-DM), and evaluate the effectiveness of an advisor (ADMIRAL-AE). We analyze the algorithms theoretically and provide fixed-point guarantees regarding their learning in general-sum stochastic games. Furthermore, extensive experiments illustrate that these algorithms: can be used in a variety of environments, have performances that compare favourably to other related baselines, can scale to large state-action spaces, and are robust to poor advice from advisors.

📄 PDF Abstract BibTeX arXiv:2111.00345

Code (1)

sriram94/multiagentadvisorqlearning 공식 구현 tf

Tasks

Decision Making Multi-agent Reinforcement LearningQ-Learningreinforcement-learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Learning from Multiple Independent Advisors in Multi-agent Reinforcement Learning

2023-01-26 · Sriram Ganapathi Subramanian, Matthew E. Taylor, Kate Larson, Mark Crowley

Multi-agent reinforcement learning typically suffers from the problem of sample inefficiency, where learning suitable policies involves the use of many data samples. Learning from external demonstrators is a possible sol…

Multi-agent Reinforcement Learning Q-Learning reinforcement-learning Reinforcement Learning+1

Scaling POMDPs For Selecting Sellers in E-markets-Extended Version

2015-11-30 · Athirai A. Irissappane, Frans A. Oliehoek, Jie Zhang

In multiagent e-marketplaces, buying agents need to select good sellers by querying other buyers (called advisors). Partially Observable Markov Decision Processes (POMDPs) have shown to be an effective framework for opti…

Agri-SAGE: Simulation-Grounded Multi-Agent LLM for Context-Aware Agricultural Advisory Generation

2026-07-01 · Vedant Balasubramaniam, Geetha Charan, Manojkumar Patil, Rohit P Suresh 외 arxiv

Agricultural advisory systems face a fundamental tension: static agronomic guidelines offer consistent, evidence-based recommendations, yet remain blind to in-season variability and dynamic uncertainties. Recent advisory…

ROSA: Roundabout Optimized Speed Advisory with Multi-Agent Trajectory Prediction in Multimodal Traffic

2026-02-16 · Anna-Lena Schlamp, Jeremias Gerner, Klaus Bogenberger, Werner Huber 외 arxiv

We present ROSA -- Roundabout Optimized Speed Advisory -- a system that combines multi-agent trajectory prediction with coordinated speed guidance for multimodal, mixed traffic at roundabouts. Using a Transformer-based m…

Trajectory Prediction

Basal-Bolus Advisor for Type 1 Diabetes (T1D) Patients Using Multi-Agent Reinforcement Learning (RL) Methodology

2023-07-17 · Mehrad Jalolia, Marzia Cescon

This paper presents a novel multi-agent reinforcement learning (RL) approach for personalized glucose control in individuals with type 1 diabetes (T1D). The method employs a closed-loop system consisting of a blood gluco…

Multi-agent Reinforcement Learning Reinforcement Learning (RL)