Interpretable Learning Dynamics in Unsupervised Reinforcement Learning

2025-05-06 · Shashwat Pandey

We present an interpretability framework for unsupervised reinforcement learning (URL) agents, aimed at understanding how intrinsic motivation shapes attention, behavior, and representation learning. We analyze five agents DQN, RND, ICM, PPO, and a Transformer-RND variant trained on procedurally generated environments, using Grad-CAM, Layer-wise Relevance Propagation (LRP), exploration metrics, and latent space clustering. To capture how agents perceive and adapt over time, we introduce two metrics: attention diversity, which measures the spatial breadth of focus, and attention change rate, which quantifies temporal shifts in attention. Our findings show that curiosity-driven agents display broader, more dynamic attention and exploratory behavior than their extrinsically motivated counterparts. Among them, TransformerRND combines wide attention, high exploration coverage, and compact, structured latent representations. Our results highlight the influence of architectural inductive biases and training signals on internal agent dynamics. Beyond reward-centric evaluation, the proposed framework offers diagnostic tools to probe perception and abstraction in RL agents, enabling more interpretable and generalizable behavior.

📄 PDF Abstract BibTeX arXiv:2505.06279

Code (0)

등록된 구현이 없습니다.

Tasks

DiagnosticDiversityreinforcement-learning Reinforcement Learning Representation LearningUnsupervised Reinforcement Learning

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Entropy Regularization 설명 없음

PPO Proximal Policy Optimization, or PPO, is a policy gradient method for reinforcement learning. The motivation was to have an algorithm with the data efficiency and reliable…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

Object-Oriented Dynamics Predictor

2018-05-25 · NeurIPS 2018 12 · Guangxiang Zhu, Zhiao Huang, Chongjie Zhang

Generalization has been one of the major challenges for learning dynamics models in model-based reinforcement learning. However, previous work on action-conditioned dynamics prediction focuses on learning the pixel-level…

Model-based Reinforcement Learning Object Reinforcement Learning

Interpretable Dynamics Models for Data-Efficient Reinforcement Learning

2019-07-10 · Markus Kaiser, Clemens Otte, Thomas Runkler, Carl Henrik Ek

In this paper, we present a Bayesian view on model-based reinforcement learning. We use expert knowledge to impose structure on the transition model and present an efficient learning scheme based on variational inference…

Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

EUCLID: Towards Efficient Unsupervised Reinforcement Learning with Multi-choice Dynamics Model

2022-10-02 · Yifu Yuan, Jianye Hao, Fei Ni, Yao Mu 외

Unsupervised reinforcement learning (URL) poses a promising paradigm to learn useful behaviors in a task-agnostic environment without the guidance of extrinsic rewards to facilitate the fast adaptation of various downstr…

reinforcement-learning Reinforcement Learning (RL)Unsupervised Pre-training Unsupervised Reinforcement Learning

SINDy-RL: Interpretable and Efficient Model-Based Reinforcement Learning

2024-03-14 · Nicholas Zolman, Urban Fasel, J. Nathan Kutz, Steven L. Brunton

Deep reinforcement learning (DRL) has shown significant promise for uncovering sophisticated control policies that interact in environments with complicated dynamics, such as stabilizing the magnetohydrodynamics of a tok…

Deep Reinforcement Learning Dictionary Learning Model-based Reinforcement Learning reinforcement-learning+1

Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning

2021-10-25 · NeurIPS 2021 12 · Jinxin Liu, Hao Shen, Donglin Wang, Yachen Kang 외

Unsupervised reinforcement learning aims to acquire skills without prior goal representations, where an agent automatically explores an open-ended environment to represent goals and learn the goal-conditioned policy. How…

Domain Adaptation reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+2