A Study of Value-Aware Eigenoptions

2025-07-12 · Harshil Kotamreddy, Marlos C. Machado arxiv

Options, which impose an inductive bias toward temporal and hierarchical structure, offer a powerful framework for reinforcement learning (RL). While effective in sequential decision-making, they are often handcrafted rather than learned. Among approaches for discovering options, eigenoptions have shown strong performance in exploration, but their role in credit assignment remains underexplored. In this paper, we investigate whether eigenoptions can accelerate credit assignment in model-free RL, evaluating them in tabular and pixel-based gridworlds. We find that pre-specified eigenoptions aid not only exploration but also credit assignment, whereas online discovery can bias the agent's experience too strongly and hinder learning. In the context of deep RL, we also propose a method for learning option-values under non-linear function approximation, highlighting the impact of termination conditions on performance. Our findings reveal both the promise and complexity of using eigenoptions, and options more broadly, to simultaneously support credit assignment and exploration in reinforcement learning.

📄 PDF Abstract BibTeX arXiv:2507.09127

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

Eigenoption Discovery through the Deep Successor Representation

2017-10-30 · ICLR 2018 1 · Marlos C. Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu 외

Options in reinforcement learning allow agents to hierarchically decompose a task into subtasks, having the potential to speed up learning and planning. However, autonomously learning effective sets of options is still a…

Atari Games Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+2

Reinforcement Learning with Options and State Representation

2024-03-16 · Ayoub Ghriss, Masashi Sugiyama, Alessandro Lazaric

The current thesis aims to explore the reinforcement learning field and build on existing methods to produce improved ones to tackle the problem of learning in high-dimensional and complex environments. It addresses such…

Decision Making Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning

Reward-Respecting Subtasks for Model-Based Reinforcement Learning

2022-02-07 · Richard S. Sutton, Marlos C. Machado, G. Zacharias Holland, David Szepesvari 외

To achieve the ambitious goals of artificial intelligence, reinforcement learning must include planning with a model of the world that is abstract in state and time. Deep learning has made progress with state abstraction…

Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

The Eigenoption-Critic Framework

2017-12-11 · Miao Liu, Marlos C. Machado, Gerald Tesauro, Murray Campbell

Eigenoptions (EOs) have been recently introduced as a promising idea for generating a diverse set of options through the graph Laplacian, having been shown to allow efficient exploration. Despite its initial promising re…

Efficient Exploration Hierarchical Reinforcement Learning Reinforcement Learning Reinforcement Learning (RL)

If They Disagree, Will You Conform? Exploring the Role of Robots' Value Awareness in a Decision-Making Task

2025-10-27 · Giulia Pusceddu, Giulio Antonio Abbo, Francesco Rea, Tony Belpaeme 외 arxiv

This study investigates whether the opinions of robotic agents can influence human decision-making when robots display value awareness (i.e., the capability of understanding human preferences and prioritizing them in dec…