Information based explanation methods for deep learning agents -- with applications on large open-source chess models

2023-09-18 · Patrik Hammersborg, Inga Strümke

With large chess-playing neural network models like AlphaZero contesting the state of the art within the world of computerised chess, two challenges present themselves: The question of how to explain the domain knowledge internalised by such models, and the problem that such models are not made openly available. This work presents the re-implementation of the concept detection methodology applied to AlphaZero in McGrath et al. (2022), by using large, open-source chess models with comparable performance. We obtain results similar to those achieved on AlphaZero, while relying solely on open-source resources. We also present a novel explainable AI (XAI) method, which is guaranteed to highlight exhaustively and exclusively the information used by the explained model. This method generates visual explanations tailored to domains characterised by discrete input spaces, as is the case for chess. Our presented method has the desirable property of controlling the information flow between any input vector and the given model, which in turn provides strict guarantees regarding what information is used by the trained model during inference. We demonstrate the viability of our method by applying it to standard 8x8 chess, using large open-source chess models.

📄 PDF Abstract BibTeX arXiv:2309.09702

Code (1)

patrik-ha/ii-map 공식 구현 tf

Methods 이 논문이 사용한 방법론

AlphaZero AlphaZero is a reinforcement learning agent for playing board games such as Go, chess, and shogi.

Integrating Policy Summaries with Reward Decomposition for Explaining Reinforcement Learning Agents

2022-10-21 · Yael Septon, Tobias Huber, Elisabeth André, Ofra Amir

Explaining the behavior of reinforcement learning agents operating in sequential decision-making settings is challenging, as their behavior is affected by a dynamic environment and delayed rewards. Methods that help user…

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Maps

2020-05-18 · Tobias Huber, Katharina Weitz, Elisabeth André, Ofra Amir

With advances in reinforcement learning (RL), agents are now being developed in high-stakes application domains such as healthcare and transportation. Explaining the behavior of these agents is challenging, as the enviro…

Atari Games Decision Making Feature Importance General Reinforcement Learning+1

Tell Me Why: Incentivizing Explanations

2025-02-19 · Siddarth Srinivasan, Ezra Karger, Michiel Bakker, YiLing Chen

Common sense suggests that when individuals explain why they believe something, we can arrive at more accurate conclusions than when they simply state what they believe. Yet, there is no known mechanism that provides inc…

Common Sense Reasoning

Understanding Individual Agent Importance in Multi-Agent System via Counterfactual Reasoning

2024-12-20 · Jianming Chen, Yawen Wang, Junjie Wang, Xiaofei Xie 외

Explaining multi-agent systems (MAS) is urgent as these systems become increasingly prevalent in various applications. Previous work has proveided explanations for the actions or states of agents, yet falls short in unde…

counterfactual Counterfactual Reasoning

ExplainBench: Evaluating Code Explanations from Agents

2026-07-29 · Zhiyuan Pan, Sungmin Kang, Imam Nur Bani Yusuf, Abhik Roychoudhury hf

Large Language Model (LLM) agents have seen rapid adoption in software engineering. As agents take a greater role in the actual generation of code, they are making larger changes, spanning tens to hundreds of lines. This…