Learning "What-if" Explanations for Sequential Decision-Making
Building interpretable parameterizations of real-world decision-making on the basis of demonstrated behavior -- i.e. trajectories of observations and actions made by an expert maximizing some unknown reward function -- is essential for introspecting and auditing policies in different institutions. In this paper, we propose learning explanations of expert decisions by modeling their reward function in terms of preferences with respect to "what if" outcomes: Given the current history of observations, what would happen if we took a particular action? To learn these cost-benefit tradeoffs associated with the expert's actions, we integrate counterfactual reasoning into batch inverse reinforcement learning. This offers a principled way of defining reward functions and explaining expert behavior, and also satisfies the constraints of real-world decision-making -- where active experimentation is often impossible (e.g. in healthcare). Additionally, by estimating the effects of different actions, counterfactuals readily tackle the off-policy nature of policy evaluation in the batch setting, and can naturally accommodate settings where the expert policies depend on histories of observations rather than just current states. Through illustrative experiments in both real and simulated medical environments, we highlight the effectiveness of our batch, counterfactual inverse reinforcement learning approach in recovering accurate and interpretable descriptions of behavior.
Code (0)
등록된 구현이 없습니다.
Tasks
counterfactualCounterfactual ReasoningDecision Makingreinforcement-learningReinforcement LearningReinforcement Learning (RL)Sequential Decision MakingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Attribution-based Explanations for Markov Decision Processes
Attribution techniques explain the outcome of an AI model by assigning a numerical score to its inputs. So far, these techniques have mainly focused on attributing importance to static input features at a single point in…
Counterfactual Explanations in Sequential Decision Making Under Uncertainty
Methods to find counterfactual explanations have predominantly focused on one step decision making processes. In this work, we initiate the development of methods to find counterfactual explanations for decision making p…
counterfactualCounterfactual ExplanationDecision MakingDecision Making Under Uncertainty+1Causal Explanations for Sequential Decision Making Under Uncertainty
We introduce a novel framework for causal explanations of stochastic, sequential decision-making systems built on the well-studied structural causal model paradigm for causal reasoning. This single framework can identify…
Causal InferenceDecision MakingDecision Making Under UncertaintySequential Decision MakingExplainable Reinforcement Learning Agents Using World Models
Explainable AI (XAI) systems have been proposed to help people understand how AI systems produce outputs and behaviors. Explainable Reinforcement Learning (XRL) has an added complexity due to the temporal nature of seque…
counterfactualreinforcement-learningReinforcement LearningSequential Decision MakingState2Explanation: Concept-Based Explanations to Benefit Agent Learning and User Understanding
As more non-AI experts use complex AI systems for daily tasks, there has been an increasing effort to develop methods that produce explanations of AI decision making that are understandable by non-AI experts. Towards thi…
Decision MakingSelf-LearningSequential Decision Making