Distributed Multi-Agent Coordination Using Multi-Modal Foundation Models

2025-01-24 · Saaduddin Mahmud, Dorian Benhamou Goldfajn, Shlomo Zilberstein

Distributed Constraint Optimization Problems (DCOPs) offer a powerful framework for multi-agent coordination but often rely on labor-intensive, manual problem construction. To address this, we introduce VL-DCOPs, a framework that takes advantage of large multimodal foundation models (LFMs) to automatically generate constraints from both visual and linguistic instructions. We then introduce a spectrum of agent archetypes for solving VL-DCOPs: from a neuro-symbolic agent that delegates some of the algorithmic decisions to an LFM, to a fully neural agent that depends entirely on an LFM for coordination. We evaluate these agent archetypes using state-of-the-art LLMs (large language models) and VLMs (vision language models) on three novel VL-DCOP tasks and compare their respective advantages and drawbacks. Lastly, we discuss how this work extends to broader frontier challenges in the DCOP literature.

📄 PDF Abstract BibTeX arXiv:2501.14189

Code (0)

등록된 구현이 없습니다.

Centralised rehearsal of decentralised cooperation: Multi-agent reinforcement learning for the scalable coordination of residential energy flexibility

2023-05-30 · Flora Charbonnier, Bei Peng, Thomas Morstyn, Malcolm McCulloch

This paper investigates how deep multi-agent reinforcement learning can enable the scalable and privacy-preserving coordination of residential energy flexibility. The coordination of distributed resources such as electri…

Multi-agent Reinforcement Learning Privacy Preserving reinforcement-learning Reinforcement Learning

MIMIC-D: Multi-modal Imitation for MultI-agent Coordination with Decentralized Diffusion Policies

2025-09-17 · Dayi Dong, Maulik Bhatt, Seoyeon Choi, Negar Mehr arxiv

As robots become more integrated in society, their ability to coordinate with other robots and humans on multi-modal tasks (those with multiple valid solutions) is crucial. Such behaviors can be learned from expert demon…

Equitable Coordination in Multi-agent Power Systems: Impacts of Computation Granularity

2023-11-20 · Yuhan Du, Javad Mohammadi

The growing integration of distributed energy resources drives the centralized power system towards a decentralized multi-agent network. Operating multi-agent networks significantly relies on inter-agent communications. …

Distributed Cooperative Multi-Agent Reinforcement Learning with Directed Coordination Graph

2022-01-10 · Gangshan Jing, He Bai, Jemin George, Aranya Chakrabortty 외

Existing distributed cooperative multi-agent reinforcement learning (MARL) frameworks usually assume undirected coordination graphs and communication graphs while estimating a global reward via consensus algorithms for p…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Silo-Bench: A Scalable Environment for Evaluating Distributed Coordination in Multi-Agent LLM Systems

2026-03-01 · Yuzhe Zhang, Feiran Liu, Yi Shan, Xinyi Huang 외 arxiv

Large language models are increasingly deployed in multi-agent systems to overcome context limitations by distributing information across agents. Yet whether agents can reliably compute with distributed information, rath…