A model-based approach to meta-Reinforcement Learning: Transformers and tree search

2022-08-24 · Brieuc Pinon, Jean-Charles Delvenne, Raphaël Jungers

Meta-learning is a line of research that develops the ability to leverage past experiences to efficiently solve new learning problems. Meta-Reinforcement Learning (meta-RL) methods demonstrate a capability to learn behaviors that efficiently acquire and exploit information in several meta-RL problems. In this context, the Alchemy benchmark has been proposed by Wang et al. [2021]. Alchemy features a rich structured latent space that is challenging for state-of-the-art model-free RL methods. These methods fail to learn to properly explore then exploit. We develop a model-based algorithm. We train a model whose principal block is a Transformer Encoder to fit the symbolic Alchemy environment dynamics. Then we define an online planner with the learned model using a tree search method. This algorithm significantly outperforms previously applied model-free RL methods on the symbolic Alchemy problem. Our results reveal the relevance of model-based approaches with online planning to perform exploration and exploitation successfully in meta-RL. Moreover, we show the efficiency of the Transformer architecture to learn complex dynamics that arise from latent spaces present in meta-RL problems.

📄 PDF Abstract BibTeX arXiv:2208.11535

Code (0)

등록된 구현이 없습니다.

Tasks

Meta-LearningMeta Reinforcement Learningreinforcement-learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Residual Connection 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Learning a Decision Tree Algorithm with Transformers

2024-02-06 · Yufan Zhuang, Liyuan Liu, Chandan Singh, Jingbo Shang 외

Decision trees are renowned for their ability to achieve high predictive performance while remaining interpretable, especially on tabular data. Traditionally, they are constructed through recursive algorithms, where they…

Meta-Learning

Hierarchical Transformers are Efficient Meta-Reinforcement Learners

2024-02-09 · Gresa Shala, André Biedenkapp, Josif Grabocka

We introduce Hierarchical Transformers for Meta-Reinforcement Learning (HTrMRL), a powerful online meta-reinforcement learning approach. HTrMRL aims to address the challenge of enabling reinforcement learning agents to p…

Meta Reinforcement Learning reinforcement-learning Reinforcement Learning

Agentic Transformers Provably Learn to Search via Reinforcement Learning

2026-05-29 · Tong Yang, Yu Huang, Yingbin Liang, Yuejie Chi arxiv

Tree search is a central abstraction behind many language-agent reasoning and decision-making tasks: agents must explore actions, remember failures, and backtrack toward promising alternatives. Yet, we lack a theoretical…

Reinforcement Learning

Transformers are Meta-Reinforcement Learners

2021-09-29 · Luckeciano Carvalho Melo

The transformer architecture and variants presented a remarkable success across many machine learning tasks in recent years. This success is intrinsically related to the capability of handling long sequences and the pres…

continuous-control Continuous Control Meta Reinforcement Learning Out-of-Distribution Generalization+3

In-context learning of evolving data streams with tabular foundational models

2025-02-24 · Afonso Lourenço, João Gama, Eric P. Xing, Goreti Marreiros

State-of-the-art data stream mining in supervised classification has traditionally relied on ensembles of incremental decision trees. However, the emergence of large tabular models, i.e., transformers designed for struct…

In-Context Learning Meta-Learning