Meta-Reinforcement Learning by Tracking Task Non-stationarity

2021-05-18 · Riccardo Poiani, Andrea Tirinzoni, Marcello Restelli

Many real-world domains are subject to a structured non-stationarity which affects the agent's goals and the environmental dynamics. Meta-reinforcement learning (RL) has been shown successful for training agents that quickly adapt to related tasks. However, most of the existing meta-RL algorithms for non-stationary domains either make strong assumptions on the task generation process or require sampling from it at training time. In this paper, we propose a novel algorithm (TRIO) that optimizes for the future by explicitly tracking the task evolution through time. At training time, TRIO learns a variational module to quickly identify latent parameters from experience samples. This module is learned jointly with an optimal exploration policy that takes task uncertainty into account. At test time, TRIO tracks the evolution of the latent parameters online, hence reducing the uncertainty over future tasks and obtaining fast adaptation through the meta-learned policy. Unlike most existing methods, TRIO does not assume Markovian task-evolution processes, it does not require information about the non-stationarity at training time, and it captures complex changes undergoing in the environment. We evaluate our algorithm on different simulated problems and show it outperforms competitive baselines.

📄 PDF Abstract BibTeX arXiv:2105.08834

Code (1)

riccardopoiani/trio-non-stationary-meta-rl 공식 구현 pytorch

Tasks

Meta Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Metatrace Actor-Critic: Online Step-size Tuning by Meta-gradient Descent for Reinforcement Learning Control

2018-05-10 · Kenny Young, Baoxiang Wang, Matthew E. Taylor

Reinforcement learning (RL) has had many successes in both "deep" and "shallow" settings. In both cases, significant hyperparameter tuning is often required to achieve good performance. Furthermore, when nonlinear functi…

Atari Games Meta-Learning Reinforcement Learning Reinforcement Learning (RL)

Safe Continual Reinforcement Learning Methods for Nonstationary Environments. Towards a Survey of the State of the Art

2026-01-08 · Timofey Tomashevskiy arxiv

This work provides a state-of-the-art survey of continual safe online reinforcement learning (COSRL) methods. We discuss theoretical aspects, challenges, and open questions in building continual online safe reinforcement…

Reinforcement Learning Continual Learning

Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning

2019-06-11 · Georgios Papoudakis, Filippos Christianos, Arrasy Rahman, Stefano V. Albrecht

Recent developments in deep reinforcement learning are concerned with creating decision-making agents which can perform well in various complex domains. A particular approach which has received increasing attention is mu…

Decision Making Deep Reinforcement Learning Meta-Learning Multi-agent Reinforcement Learning+3

MetaCURL: Non-stationary Concave Utility Reinforcement Learning

2024-05-30 · Bianca Marin Moreno, Margaux Brégère, Pierre Gaillard, Nadia Oudjane

We explore online learning in episodic loop-free Markov decision processes on non-stationary environments (changing losses and probability transitions). Our focus is on the Concave Utility Reinforcement Learning problem …

reinforcement-learning Reinforcement Learning

Stable Deep Reinforcement Learning via Isotropic Gaussian Representations

2026-02-22 · Ali Saheb Pasand, Johan Obando-Ceron, Aaron Courville, Pouya Bashivan 외 arxiv

Deep reinforcement learning systems often suffer from unstable training dynamics due to non-stationarity, where learning objectives and data distributions evolve over time. We show that under non-stationary targets, isot…

Reinforcement Learning