Explainable Multi-Agent Reinforcement Learning for Temporal Queries
As multi-agent reinforcement learning (MARL) systems are increasingly deployed throughout society, it is imperative yet challenging for users to understand the emergent behaviors of MARL agents in complex environments. This work presents an approach for generating policy-level contrastive explanations for MARL to answer a temporal user query, which specifies a sequence of tasks completed by agents with possible cooperation. The proposed approach encodes the temporal query as a PCTL logic formula and checks if the query is feasible under a given MARL policy via probabilistic model checking. Such explanations can help reconcile discrepancies between the actual and anticipated multi-agent behaviors. The proposed approach also generates correct and complete explanations to pinpoint reasons that make a user query infeasible. We have successfully applied the proposed approach to four benchmark MARL domains (up to 9 agents in one domain). Moreover, the results of a user study show that the generated explanations significantly improve user performance and satisfaction.
Code (1)
Tasks
Multi-agent Reinforcement Learningreinforcement-learningReinforcement LearningSimilar Papers 제목 키워드 기반
ASQ-IT: Interactive Explanations for Reinforcement-Learning Agents
As reinforcement learning methods increasingly amass accomplishments, the need for comprehending their solutions becomes more crucial. Most explainable reinforcement learning (XRL) methods generate a static explanation d…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Toward Policy Explanations for Multi-Agent Reinforcement Learning
Advances in multi-agent reinforcement learning (MARL) enable sequential decision making for a range of exciting multi-agent applications such as cooperative AI and autonomous driving. Explaining agent decisions is crucia…
Autonomous DrivingDecision MakingHuman Agent CollaborationMulti-agent Reinforcement Learning+4TalkToAgent: A Human-centric Explanation of Reinforcement Learning Agents with Large Language Models
Explainable Reinforcement Learning (XRL) has emerged as a promising approach in improving the transparency of Reinforcement Learning (RL) agents. However, there remains a gap between complex RL policies and domain expert…
Reinforcement LearningBXRL: Behavior-Explainable Reinforcement Learning
A major challenge of Reinforcement Learning is that agents often learn undesired behaviors that seem to defy the reward structure they were given. Explainable Reinforcement Learning (XRL) methods can answer queries such …
Reinforcement LearningExplainable Reinforcement Learning Agents Using World Models
Explainable AI (XAI) systems have been proposed to help people understand how AI systems produce outputs and behaviors. Explainable Reinforcement Learning (XRL) has an added complexity due to the temporal nature of seque…
counterfactualreinforcement-learningReinforcement LearningSequential Decision Making