Interpreting Reinforcement Policies through Local Behaviors
Many works in explainable AI have focused on explaining black-box classification models. Explaining deep reinforcement learning (RL) policies in a manner that could be understood by domain users has received much less attention. In this paper, we propose a novel perspective to understanding RL policies based on identifying important states from automatically learned meta-states. The key conceptual difference between our approach and many previous ones is that we form meta-states based on locality governed by the expert policy dynamics rather than based on similarity of actions, and that we do not assume any particular knowledge of the underlying topology of the state space. Theoretically, we show that our algorithm to find meta-states converges and the objective that selects important states from each meta-state is submodular leading to efficient high quality greedy selection. Experiments on three domains (four rooms, door-key and minipacman) and a carefully conducted user study illustrate that our perspective leads to better understanding of the policy. We conjecture that this is a result of our meta-states being more intuitive in that the corresponding important states are strong indicators of tractable intermediate goals that are easier for humans to interpret and follow.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep Reinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Imagination-Augmented Hierarchical Reinforcement Learning for Safe and Interactive Autonomous Driving in Urban Environments
Hierarchical reinforcement learning (HRL) incorporates temporal abstraction into reinforcement learning (RL) by explicitly taking advantage of hierarchical structure. Modern HRL typically designs a hierarchical agent com…
Autonomous DrivingHierarchical Reinforcement LearningReinforcement Learning (RL)FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Data
Offline Federated Reinforcement Learning (FRL), a marriage of federated learning and offline reinforcement learning, has attracted increasing interest recently. Albeit with some advancement, we find that the performance …
Reinforcement LearningFederated LearningLearning Navigation Behaviors End-to-End with AutoRL
We learn end-to-end point-to-point and path-following navigation behaviors that avoid moving obstacles. These policies receive noisy lidar observations and output robot linear and angular velocities. The policies are tra…
Deep Reinforcement LearningMotion Planningreinforcement-learningReinforcement Learning+1Physics-Informed Modeling and Control of Emergent Behaviors in Robot Swarms
Robot swarms can exhibit coherent collective behaviors through local perception, limited communication and decentralized decision-making, yet modeling and controlling such emergence remains challenging when behaviors unf…
Reinforcement LearningDiversity Through Exclusion (DTE): Niche Identification for Reinforcement Learning through Value-Decomposition
Many environments contain numerous available niches of variable value, each associated with a different local optimum in the space of behaviors (policy space). In such situations it is often difficult to design a learnin…
DiversityQ-Learningreinforcement-learningReinforcement Learning (RL)