Benchmarking Perturbation-based Saliency Maps for Explaining Atari Agents

2021-01-18 · Tobias Huber, Benedikt Limmer, Elisabeth André

One of the most prominent methods for explaining the behavior of Deep Reinforcement Learning (DRL) agents is the generation of saliency maps that show how much each pixel attributed to the agents' decision. However, there is no work that computationally evaluates and compares the fidelity of different saliency map approaches specifically for DRL agents. It is particularly challenging to computationally evaluate saliency maps for DRL agents since their decisions are part of an overarching policy. For instance, the output neurons of value-based DRL algorithms encode both the value of the current state as well as the value of doing each action in this state. This ambiguity should be considered when evaluating saliency maps for such agents. In this paper, we compare five popular perturbation-based approaches to create saliency maps for DRL agents trained on four different Atari 2600 games. The approaches are compared using two computational metrics: dependence on the learned parameters of the agent (sanity checks) and fidelity to the agent's reasoning (input degradation). During the sanity checks, we encounter issues with one approach and propose a solution to fix these issues. For fidelity, we identify two main factors that influence which saliency approach should be chosen in which situation.

📄 PDF Abstract BibTeX arXiv:2101.07312

Code (1)

belimmer/PerturbationSaliencyEvaluation 공식 구현 tf

Tasks

Atari Games BenchmarkingDeep Reinforcement LearningFeature Importance Image Classification

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

Visualizing and Understanding Atari Agents

2017-10-31 · ICML 2018 7 · Sam Greydanus, Anurag Koul, Jonathan Dodge, Alan Fern

While deep reinforcement learning (deep RL) agents are effective at maximizing rewards, it is often unclear what strategies they use to do so. In this paper, we take a step toward explaining deep RL agents through a case…

Deep Reinforcement Learning Reinforcement Learning Reinforcement Learning (RL)

Explain Your Move: Understanding Agent Actions Using Focused Feature Saliency

2020-05-01 · ICLR 2020 1 · Piyush Gupta, Nikaash Puri, Sukriti Verma, Dhruv Kayastha 외

As deep reinforcement learning (RL) is applied to more tasks, there is a need to visualize and understand the behavior of learned agents. Saliency maps explain agent behavior by highlighting the features of the input sta…

Atari Games Board Games Deep Reinforcement Learning Reinforcement Learning (RL)+1

Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attribution

2019-12-23 · Nikaash Puri, Sukriti Verma, Piyush Gupta, Dhruv Kayastha 외

Atari Games Board Games Deep Reinforcement Learning Reinforcement Learning+2

Saliency Methods for Explaining Adversarial Attacks

2019-08-22 · Jindong Gu, Volker Tresp

The classification decisions of neural networks can be misled by small imperceptible perturbations. This work aims to explain the misled classifications using saliency methods. The idea behind saliency methods is to expl…

General Classification

Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning

2019-12-09 · ICLR 2020 1 · Akanksha Atrey, Kaleigh Clary, David Jensen

Saliency maps are frequently used to support explanations of the behavior of deep reinforcement learning (RL) agents. However, a review of how saliency maps are used in practice indicates that the derived explanations ar…

Atari Games counterfactual Counterfactual Reasoning Deep Reinforcement Learning+3