Debiased Offline Representation Learning for Fast Online Adaptation in Non-stationary Dynamics

2024-02-17 · Xinyu Zhang, Wenjie Qiu, Yi-Chen Li, Lei Yuan, Chengxing Jia, Zongzhang Zhang, Yang Yu

Developing policies that can adjust to non-stationary environments is essential for real-world reinforcement learning applications. However, learning such adaptable policies in offline settings, with only a limited set of pre-collected trajectories, presents significant challenges. A key difficulty arises because the limited offline data makes it hard for the context encoder to differentiate between changes in the environment dynamics and shifts in the behavior policy, often leading to context misassociations. To address this issue, we introduce a novel approach called Debiased Offline Representation for fast online Adaptation (DORA). DORA incorporates an information bottleneck principle that maximizes mutual information between the dynamics encoding and the environmental data, while minimizing mutual information between the dynamics encoding and the actions of the behavior policy. We present a practical implementation of DORA, leveraging tractable bounds of the information bottleneck principle. Our experimental evaluation across six benchmark MuJoCo tasks with variable parameters demonstrates that DORA not only achieves a more precise dynamics encoding but also significantly outperforms existing baselines in terms of performance.

📄 PDF Abstract BibTeX arXiv:2402.11317

Code (1)

lamda-rl/dora 공식 구현 pytorch

Tasks

MuJoCoRepresentation Learning

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

PAnDR: Fast Adaptation to New Environments from Offline Experiences via Decoupling Policy and Environment Representations

2022-04-06 · Tong Sang, Hongyao Tang, Yi Ma, Jianye Hao 외

Deep Reinforcement Learning (DRL) has been a promising solution to many complex decision-making problems. Nevertheless, the notorious weakness in generalization among environments prevent widespread application of DRL ag…

Contrastive Learning Decision Making Deep Reinforcement Learning

Ensemble Successor Representations for Task Generalization in Offline-to-Online Reinforcement Learning

2024-05-12 · Changhong Wang, Xudong Yu, Chenjia Bai, Qiaosheng Zhang 외

In Reinforcement Learning (RL), training a policy from scratch with online experiences can be inefficient because of the difficulties in exploration. Recently, offline RL provides a promising solution by giving an initia…

Offline RL Reinforcement Learning (RL)Representation Learning

Offline Meta Reinforcement Learning with In-Distribution Online Adaptation

2023-05-31 · Jianhao Wang, Jin Zhang, Haozhe Jiang, Junyu Zhang 외

Recent offline meta-reinforcement learning (meta-RL) methods typically utilize task-dependent behavior policies (e.g., training RL agents on each individual task) to collect a multi-task dataset. However, these methods a…

Meta Reinforcement Learning reinforcement-learning Reinforcement Learning Uncertainty Quantification

MOORe: Model-based Offline-to-Online Reinforcement Learning

2022-01-25 · Yihuan Mao, Chao Wang, Bin Wang, Chongjie Zhang

With the success of offline reinforcement learning (RL), offline trained RL policies have the potential to be further improved when deployed online. A smooth transfer of the policy matters in safe real-world deployment. …

D4RL model reinforcement-learning Reinforcement Learning+1

Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL

2025-11-05 · Lipeng Zu, Hansong Zhou, Xiaonan Zhang arxiv

Offline reinforcement learning (RL) enables training from fixed data without online interaction, but policies learned offline often struggle when deployed in dynamic environments due to distributional shift and unreliabl…

Reinforcement Learning