홈 › Papers

Mechanism Design Is Not Enough: Prosocial Agents for Cooperative AI

2026-05-08 · Xuanqiang Angelo Huang, Charlie Tharas, Samuele Marro, Van Q. Truong, Bernhard Schölkopf, Emanuele La Malfa, Zhijing Jin arxiv

Ensuring that AI agents behave safely and beneficially when interacting with other parties has emerged as one of the central challenges of modern AI safety. While mechanism design, as the theory of designing rules to align individual and collective objectives, can incentivize cooperative behavior, it is still an open question whether it alone is sufficient to maximize LLM agents' social welfare. This work proves that the answer is negative: drawing from incomplete contract theory, we formally show that when contracts cannot distinguish all relevant future contingencies, there is a strictly positive welfare loss that no realistic mechanism can eliminate. We show that prosocial agents, who weigh others' welfare alongside their own, can close this gap and achieve outcomes that are socially superior and individually beneficial. Experimentally, we show that in multi-agent resource-allocation environments and canonical social dilemmas where agents are powered by large language models, prosociality is beneficial. The implication for AI safety is clear: to enable cooperative interactions at scale, designing adequate mechanisms is not sufficient; agents must be built to be intrinsically prosocial.

📄 PDF Abstract BibTeX arXiv:2605.08426

Code (0)

등록된 구현이 없습니다.

Studying the Impact of Semi-Cooperative Drivers on Overall Highway Flow

2023-04-23 · Noam Buckman, Sertac Karaman, Daniela Rus

Semi-cooperative behaviors are intrinsic properties of human drivers and should be considered for autonomous driving. In addition, new autonomous planners can consider the social value orientation (SVO) of human drivers …

Autonomous Driving

Partner Selection for the Emergence of Cooperation in Multi-Agent Systems Using Reinforcement Learning

2019-02-08 · Nicolas Anastassacos, Stephen Hailes, Mirco Musolesi

Social dilemmas have been widely studied to explain how humans are able to cooperate in society. Considerable effort has been invested in designing artificial agents for social dilemmas that incorporate explicit agent mo…

Multi-agent Reinforcement Learning Reinforcement Learning

Emergent Prosociality in Multi-Agent Games Through Gifting

2021-05-13 · Woodrow Z. Wang, Mark Beliaev, Erdem Biyik, Daniel A. Lazar 외

Coordination is often critical to forming prosocial behaviors -- behaviors that increase the overall sum of rewards received by all agents in a multi-agent game. However, state of the art reinforcement learning algorithm…

Emergent Communication through Negotiation

2018-04-11 · ICLR 2018 1 · Kris Cao, Angeliki Lazaridou, Marc Lanctot, Joel Z. Leibo 외

Multi-agent reinforcement learning offers a way to study how communication could emerge in communities of agents needing to solve specific problems. In this paper, we study the emergence of communication in the negotiati…

Multi-agent Reinforcement Learning Reinforcement Learning

Prosocial learning agents solve generalized Stag Hunts better than selfish ones

2017-09-08 · Alexander Peysakhovich, Adam Lerer

Deep reinforcement learning has become an important paradigm for constructing agents that can enter complex multi-agent situations and improve their policies through experience. One commonly used technique is reactive tr…

Deep Reinforcement Learning Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning+1