Offline Reinforcement Learning from Datasets with Structured Non-Stationarity

2024-05-23 · Johannes Ackermann, Takayuki Osa, Masashi Sugiyama

Current Reinforcement Learning (RL) is often limited by the large amount of data needed to learn a successful policy. Offline RL aims to solve this issue by using transitions collected by a different behavior policy. We address a novel Offline RL problem setting in which, while collecting the dataset, the transition and reward functions gradually change between episodes but stay constant within each episode. We propose a method based on Contrastive Predictive Coding that identifies this non-stationarity in the offline dataset, accounts for it when training a policy, and predicts it during evaluation. We analyze our proposed method and show that it performs well in simple continuous control tasks and challenging, high-dimensional locomotion tasks. We show that our method often achieves the oracle performance and performs better than baselines.

📄 PDF Abstract BibTeX arXiv:2405.14114

Code (1)

johannesack/offlinerlstructurednonstationarity 공식 구현 jax

Tasks

continuous-control Continuous Control Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

InfoNCE 설명 없음

Contrastive Predictive Coding Contrastive Predictive Coding (CPC) learns self-supervised representations by predicting the future in latent space by using powerful autoregressive models. The model uses a…

Forecasting in Offline Reinforcement Learning for Non-stationary Environments

2025-12-01 · Suzan Ece Ada, Georg Martius, Emre Ugur, Erhan Oztop arxiv

Offline Reinforcement Learning (RL) provides a promising avenue for training policies from pre-collected datasets when gathering additional interaction data is infeasible. However, existing offline RL methods often assum…

Reinforcement Learning Offline RL

Testing Stationarity and Change Point Detection in Reinforcement Learning

2022-03-03 · Mengbing Li, Chengchun Shi, Zhenke Wu, Piotr Fryzlewicz

We consider offline reinforcement learning (RL) methods in possibly nonstationary environments. Many existing RL algorithms in the literature rely on the stationarity assumption that requires the system transition and th…

Change Point Detection reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Off-the-Grid MARL: Datasets with Baselines for Offline Multi-Agent Reinforcement Learning

2023-02-01 · Claude Formanek, Asad Jeewa, Jonathan Shock, Arnu Pretorius

Being able to harness the power of large datasets for developing cooperative multi-agent controllers promises to unlock enormous value for real-world applications. Many important industrial systems are multi-agent in nat…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)

Safety-aware Causal Representation for Trustworthy Offline Reinforcement Learning in Autonomous Driving

2023-10-31 · Haohong Lin, Wenhao Ding, Zuxin Liu, Yaru Niu 외

In the domain of autonomous driving, the offline Reinforcement Learning~(RL) approaches exhibit notable efficacy in addressing sequential decision-making problems from offline datasets. However, maintaining safety in div…

Autonomous Driving Autonomous Vehicles Decision Making Offline RL+4

Puzzle it Out: Local-to-Global World Model for Offline Multi-Agent Reinforcement Learning

2026-01-12 · Sijia Li, Xinran Li, Shibo Chen, Jun Zhang arxiv

Offline multi-agent reinforcement learning (MARL) aims to solve cooperative decision-making problems in multi-agent systems using pre-collected datasets. Existing offline MARL methods primarily constrain training within …

Multi-agent Reinforcement Learning