TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraft

2024-12-06 · Qian Long, Zhi Li, Ran Gong, Ying Nian Wu, Demetri Terzopoulos, Xiaofeng Gao

Collaboration is a cornerstone of society. In the real world, human teammates make use of multi-sensory data to tackle challenging tasks in ever-changing environments. It is essential for embodied agents collaborating in visually-rich environments replete with dynamic interactions to understand multi-modal observations and task specifications. To evaluate the performance of generalizable multi-modal collaborative agents, we present TeamCraft, a multi-modal multi-agent benchmark built on top of the open-world video game Minecraft. The benchmark features 55,000 task variants specified by multi-modal prompts, procedurally-generated expert demonstrations for imitation learning, and carefully designed protocols to evaluate model generalization capabilities. We also perform extensive analyses to better understand the limitations and strengths of existing approaches. Our results indicate that existing models continue to face significant challenges in generalizing to novel goals, scenes, and unseen numbers of agents. These findings underscore the need for further research in this area. The TeamCraft platform and dataset are publicly available at https://github.com/teamcraft-bench/teamcraft.

📄 PDF Abstract BibTeX arXiv:2412.05255

Code (1)

teamcraft-bench/teamcraft 공식 구현 pytorch

Tasks

Imitation LearningMinecraft

CH-MARL: A Multimodal Benchmark for Cooperative, Heterogeneous Multi-Agent Reinforcement Learning

2022-08-26 · Vasu Sharma, Prasoon Goyal, Kaixiang Lin, Govind Thattai 외

We propose a multimodal (vision-and-language) benchmark for cooperative and heterogeneous multi-agent learning. We introduce a benchmark multimodal dataset with tasks involving collaboration between multiple simulated he…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

InterLV-Search: Benchmarking Interleaved Multimodal Agentic Search

2026-05-08 · Bohan Hou, Jiuning Gu, Jiayan Guo, Ronghao Dang 외 arxiv

Existing benchmarks for multimodal agentic search evaluate multimodal search and visual browsing, but visual evidence is either confined to the input or treated as an answer endpoint rather than part of an interleaved se…

M$^3$Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions

2026-06-05 · Zhengjun Huang, Wenxuan Liu, Zhoujin Tian, Wei Chen 외 arxiv

Language agents are increasingly deployed over accumulating multimodal information, yet existing benchmarks assume a human-human form with sparse visuals and straightforward content, evaluating neither reasoning over aut…

SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory

2026-05-15 · Huacan Chai, Yukai Wang, Yingxuan Yang, Dan Peng 외 arxiv

Existing benchmarks for multimodal memory reasoning largely evaluate systems within pre-assembled contexts, but under-evaluate whether agents can use evidence distributed across independently originated sources. We argue…

Multimodal Reasoning

AgentRx: A Benchmark Study of LLM Agents for Multimodal Clinical Prediction Tasks

2026-05-11 · Baraa Al Jorf, Farah E. Shamout arxiv

Building effective clinical decision support systems requires the synthesis of complex heterogeneous multimodal data. Such modalities include temporal electronic health records data, medical images, radiology reports, an…