Discovering Causality for Efficient Cooperation in Multi-Agent Environments
In cooperative Multi-Agent Reinforcement Learning (MARL) agents are required to learn behaviours as a team to achieve a common goal. However, while learning a task, some agents may end up learning sub-optimal policies, not contributing to the objective of the team. Such agents are called lazy agents due to their non-cooperative behaviours that may arise from failing to understand whether they caused the rewards. As a consequence, we observe that the emergence of cooperative behaviours is not necessarily a byproduct of being able to solve a task as a team. In this paper, we investigate the applications of causality in MARL and how it can be applied in MARL to penalise these lazy agents. We observe that causality estimations can be used to improve the credit assignment to the agents and show how it can be leveraged to improve independent learning in MARL. Furthermore, we investigate how Amortized Causal Discovery can be used to automate causality detection within MARL environments. The results demonstrate that causality relations between individual observations and the team reward can be used to detect and punish lazy agents, making them develop more intelligent behaviours. This results in improvements not only in the overall performances of the team but also in their individual capabilities. In addition, results show that Amortized Causal Discovery can be used efficiently to find causal relations in MARL.
Code (1)
Tasks
Causal DiscoveryMulti-agent Reinforcement LearningSimilar Papers 제목 키워드 기반
Learning Independently from Causality in Multi-Agent Environments
Multi-Agent Reinforcement Learning (MARL) comprises an area of growing interest in the field of machine learning. Despite notable advances, there are still problems that require investigation. The lazy agent pathology is…
Multi-agent Reinforcement LearningRelationCaPo: Cooperative Plan Optimization for Efficient Embodied Multi-Agent Cooperation
In this work, we address the cooperation problem among large language model (LLM) based embodied agents, where agents must cooperate to achieve a common goal. Previous methods often execute actions extemporaneously and i…
Large Language ModelUnderstanding the World to Solve Social Dilemmas Using Multi-Agent Reinforcement Learning
Social dilemmas are situations where groups of individuals can benefit from mutual cooperation but conflicting interests impede them from doing so. This type of situations resembles many of humanity's most critical chall…
Multi-agent Reinforcement Learningreinforcement-learningReinforcement Learning (RL)Emergent Cooperation under Uncertain Incentive Alignment
Understanding the emergence of cooperation in systems of computational agents is crucial for the development of effective cooperative AI. Interaction among individuals in real-world settings are often sparse and occur wi…
Causality Detection for Efficient Multi-Agent Reinforcement Learning
When learning a task as a team, some agents in Multi-Agent Reinforcement Learning (MARL) may fail to understand their true impact in the performance of the team. Such agents end up learning sub-optimal policies, demonstr…
Multi-agent Reinforcement Learningreinforcement-learningReinforcement Learning