Explainable Reinforcement Learning Agents Using World Models

2025-05-12 · Madhuri Singh, Amal Alabdulkarim, Gennie Mansi, Mark O. Riedl

Explainable AI (XAI) systems have been proposed to help people understand how AI systems produce outputs and behaviors. Explainable Reinforcement Learning (XRL) has an added complexity due to the temporal nature of sequential decision-making. Further, non-AI experts do not necessarily have the ability to alter an agent or its policy. We introduce a technique for using World Models to generate explanations for Model-Based Deep RL agents. World Models predict how the world will change when actions are performed, allowing for the generation of counterfactual trajectories. However, identifying what a user wanted the agent to do is not enough to understand why the agent did something else. We augment Model-Based RL agents with a Reverse World Model, which predicts what the state of the world should have been for the agent to prefer a given counterfactual action. We show that explanations that show users what the world should have been like significantly increase their understanding of the agent policy. We hypothesize that our explanations can help users learn how to control the agents execution through by manipulating the environment.

📄 PDF Abstract BibTeX arXiv:2505.08073

Code (0)

등록된 구현이 없습니다.

Tasks

counterfactualreinforcement-learning Reinforcement LearningSequential Decision Making

Complementary reinforcement learning towards explainable agents

2019-01-01 · Jung Hoon Lee

Reinforcement learning (RL) algorithms allow agents to learn skills and strategies to perform complex tasks without detailed instructions or expensive labelled training examples. That is, RL agents can learn, as we learn…

Decision Making Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Explainable Reinforcement Learning Through a Causal Lens

2019-05-27 · Prashan Madumal, Tim Miller, Liz Sonenberg, Frank Vetere

Prevalent theories in cognitive science propose that humans understand and represent the knowledge of the world through causal relationships. In making sense of the world, we build causal models in our mind to encode cau…

counterfactual reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+2

End-to-End Policy Gradient Method for POMDPs and Explainable Agents

2023-04-19 · Soichiro Nishimori, Sotetsu Koyamada, Shin Ishii

Real-world decision-making problems are often partially observable, and many can be formulated as a Partially Observable Markov Decision Process (POMDP). When we apply reinforcement learning (RL) algorithms to the POMDP,…

Autonomous Driving Decision Making reinforcement-learning Reinforcement Learning (RL)

A Survey of Explainable Reinforcement Learning

2022-02-17 · Stephanie Milani, Nicholay Topin, Manuela Veloso, Fei Fang

Explainable reinforcement learning (XRL) is an emerging subfield of explainable machine learning that has attracted considerable attention in recent years. The goal of XRL is to elucidate the decision-making process of l…

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+2

Explainable Deep Reinforcement Learning: State of the Art and Challenges

2023-01-24 · George A. Vouros

Interpretability, explainability and transparency are key issues to introducing Artificial Intelligence methods in many critical domains: This is important due to ethical concerns and trust issues strongly connected to r…

Decision Making Deep Reinforcement Learning Fairness reinforcement-learning+2