State2Explanation: Concept-Based Explanations to Benefit Agent Learning and User Understanding

2023-09-21 · NeurIPS 2023 11

As more non-AI experts use complex AI systems for daily tasks, there has been an increasing effort to develop methods that produce explanations of AI decision making that are understandable by non-AI experts. Towards this effort, leveraging higher-level concepts and producing concept-based explanations have become a popular method. Most concept-based explanations have been developed for classification techniques, and we posit that the few existing methods for sequential decision making are limited in scope. In this work, we first contribute a desiderata for defining concepts in sequential decision making settings. Additionally, inspired by the Protege Effect which states explaining knowledge often reinforces one's self-learning, we explore how concept-based explanations of an RL agent's decision making can in turn improve the agent's learning rate, as well as improve end-user understanding of the agent's decision making. To this end, we contribute a unified framework, State2Explanation (S2E), that involves learning a joint embedding model between state-action pairs and concept-based explanations, and leveraging such learned model to both (1) inform reward shaping during an agent's training, and (2) provide explanations to end-users at deployment for improved task performance. Our experimental validations, in Connect 4 and Lunar Lander, demonstrate the success of S2E in providing a dual-benefit, successfully informing reward shaping and improving agent learning rate, as well as significantly improving end user task performance at deployment time.

📄 PDF Abstract BibTeX arXiv:2309.12482

Code (0)

등록된 구현이 없습니다.

Tasks

Decision MakingSelf-LearningSequential Decision Making

DEAL: Disentangle and Localize Concept-level Explanations for VLMs

2024-07-19 · Tang Li, Mengmeng Ma, Xi Peng

Large pre-trained Vision-Language Models (VLMs) have become ubiquitous foundational components of other models and downstream tasks. Although powerful, our empirical results reveal that such models might not be able to i…

Integrating Policy Summaries with Reward Decomposition for Explaining Reinforcement Learning Agents

2022-10-21 · Yael Septon, Tobias Huber, Elisabeth André, Ofra Amir

Explaining the behavior of reinforcement learning agents operating in sequential decision-making settings is challenging, as their behavior is affected by a dynamic environment and delayed rewards. Methods that help user…

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomes

2023-12-18 · Yotam Amitai, Yael Septon, Ofra Amir

Explainable reinforcement learning (XRL) methods aim to help elucidate agent policies and decision-making processes. The majority of XRL approaches focus on local explanations, seeking to shed light on the reasons an age…

counterfactual Decision Making reinforcement-learning Reinforcement Learning

Tell me why! Explanations support learning relational and causal structure

2021-12-07 · Andrew K. Lampinen, Nicholas A. Roy, Ishita Dasgupta, Stephanie C. Y. Chan 외

Inferring the abstract relational and causal structure of the world is a major challenge for reinforcement-learning (RL) agents. For humans, language--particularly in the form of explanations--plays a considerable role i…

Odd One Out Reinforcement Learning (RL)

Not all users are the same: Providing personalized explanations for sequential decision making problems

2021-06-23 · Utkarsh Soni, Sarath Sreedharan, Subbarao Kambhampati

There is a growing interest in designing autonomous agents that can work alongside humans. Such agents will undoubtedly be expected to explain their behavior and decisions. While generating explanations is an actively re…

All Clustering Decision Making Explanation Generation+1