In-Context Reinforcement Learning for Variable Action Spaces

2023-12-20 · Viacheslav Sinii, Alexander Nikulin, Vladislav Kurenkov, Ilya Zisman, Sergey Kolesnikov

Recently, it has been shown that transformers pre-trained on diverse datasets with multi-episode contexts can generalize to new reinforcement learning tasks in-context. A key limitation of previously proposed models is their reliance on a predefined action space size and structure. The introduction of a new action space often requires data re-collection and model re-training, which can be costly for some applications. In our work, we show that it is possible to mitigate this issue by proposing the Headless-AD model that, despite being trained only once, is capable of generalizing to discrete action spaces of variable size, semantic content and order. By experimenting with Bernoulli and contextual bandits, as well as a gridworld environment, we show that Headless-AD exhibits significant capability to generalize to action spaces it has never encountered, even outperforming specialized models trained for a specific set of actions on several environment configurations. Implementation is available at: https://github.com/corl-team/headless-ad.

📄 PDF Abstract BibTeX arXiv:2312.13327

Code (1)

corl-team/headless-ad 공식 구현 pytorch

Tasks

In-Context Reinforcement LearningMulti-Armed Bandits reinforcement-learning Reinforcement Learning

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models

2019-02-23 · NAACL 2019 6 · Tiancheng Zhao, Kaige Xie, Maxine Eskenazi

Defining action spaces for conversational agents and optimizing their decision-making process with reinforcement learning is an enduring challenge. Common practice has been to use handcrafted dialog acts, or the output v…

Decision Making Dialogue Generation Dialogue Management Goal-Oriented Dialogue Systems+3

Deep Reinforcement Learning in Parameterized Action Space

2015-11-13 · Matthew Hausknecht, Peter Stone

Recent work has shown that deep neural networks are capable of approximating both value functions and policies in reinforcement learning domains featuring continuous state and action spaces. However, to the best of our k…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Learning Multi-Timescale Abstractions for Hierarchical Combinatorial Planning

2026-05-16 · Vivienne Huiling Wang, Tinghuai Wang, Joni Pajarinen arxiv

The combination of exponentially large action spaces, stochastic dynamics, and long-horizon decision-making under limited resources makes Sequential Stochastic Combinatorial Optimization (SSCO) particularly challenging f…

Hierarchical Reinforcement Learning

LAVA: Latent Action Spaces via Variational Auto-encoding for Dialogue Policy Optimization

2020-11-18 · COLING 2020 8 · Nurul Lubis, Christian Geishauser, Michael Heck, Hsien-Chin Lin 외

Reinforcement learning (RL) can enable task-oriented dialogue systems to steer the conversation towards successful task completion. In an end-to-end setting, a response can be constructed in a word-level sequential decis…

Decision Making Reinforcement Learning (RL)Sequential Decision Making Task-Oriented Dialogue Systems

Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks

2019-06-20 · Roberto Martín-Martín, Michelle A. Lee, Rachel Gardner, Silvio Savarese 외

Reinforcement Learning (RL) of contact-rich manipulation tasks has yielded impressive results in recent years. While many studies in RL focus on varying the observation space or reward model, few efforts focused on the c…

Contact-rich Manipulation Reinforcement Learning Reinforcement Learning (RL)