Emergent Collusion in Long-Horizon LLM Agent Interaction

2026-09-21 · Xinrui Shi, Yanzhe Zhang, Diyi Yang hf

LLM agents are increasingly deployed in collaborative settings, yet long-term interaction may give rise to undesirable coordination. We study the emergence of collusion in a long-horizon multi-agent environment: two agents repeatedly complete individual tasks, share task logs, verify each other's work, and receive rewards. We introduce realistic constraints that make compliance with the verification protocol incompatible with reward maximization, and find that agents increasingly deviate from the protocol over repeated interactions. Collusion emerges in 94% of trajectories across 10 models, and more capable models within the same family reach it earlier. Controlled peer interventions show that collusion is shaped by peer behavior, while ablations reveal additional effects of reward structure, the verification feedback agents receive, and their interaction history. In particular, restricting the amount and scope of interaction history available to agents reduces collusion. Overall, our findings show that long-horizon interaction can reshape how agents coordinate in ways that create safety risks.

📄 PDF Abstract BibTeX arXiv:2609.24967

Code (2)

SALT-NLP/agent-collusion ★ 8

Valiant-Cat/hfpaper

Multi-Agent Reinforcement Learning for Market Making: Competition without Collusion

2025-10-29 · Ziyi Wang, Carmine Ventre, Maria Polukarov arxiv

Algorithmic collusion has emerged as a central question in AI: Will the interaction between different AI agents deployed in markets lead to collusion? More generally, understanding how emergent behavior, be it a cartel o…

Multi-agent Reinforcement Learning

Mapping Human Anti-collusion Mechanisms to Multi-agent AI Systems

2026-01-01 · Jamiu Idowu, Ahmed Almasoud, Ayman Alfahid arxiv

As multi-agent AI systems become increasingly autonomous, evidence shows they can develop collusive strategies similar to those long observed in human markets and institutions. While human domains have accumulated centur…

Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMs

2024-10-02 · Yohan Mathew, Ollie Matthews, Robert McCarthy, Joan Velja 외

The rapid proliferation of frontier model agents promises significant societal advances but also raises concerns about systemic risks arising from unsafe interactions. Collusion to the disadvantage of others has been ide…

In-Context Reinforcement Learning reinforcement-learning Reinforcement Learning

AI agents in Algorithmic Electricity Markets: On the Emergence of Tacit Collusion

2026-08-27 · Jakub Seredyński, Georgios Tsaousoglou arxiv

As electricity market participants increasingly adopt learning-based agents for their bidding strategies, electricity markets are becoming algorithmic. Evidence from algorithmic markets in other domains shows that tacit …

Multi-agent Reinforcement Learning

Colosseum: Auditing Collusion in Cooperative Multi-Agent Systems

2026-02-16 · Mason Nakamura, Abhinav Kumar, Saswat Das, Sahar Abdelnabi 외 arxiv

Multi-agent systems, where LLM agents communicate through free-form language, enable sophisticated coordination for solving complex cooperative tasks. This surfaces a unique safety problem when a group of agents forms a …