Continual Task Learning through Adaptive Policy Self-Composition

2024-11-18 · Shengchao Hu, YuHang Zhou, Ziqing Fan, Jifeng Hu, Li Shen, Ya zhang, DaCheng Tao

Training a generalizable agent to continually learn a sequence of tasks from offline trajectories is a natural requirement for long-lived agents, yet remains a significant challenge for current offline reinforcement learning (RL) algorithms. Specifically, an agent must be able to rapidly adapt to new tasks using newly collected trajectories (plasticity), while retaining knowledge from previously learned tasks (stability). However, systematic analyses of this setting are scarce, and it remains unclear whether conventional continual learning (CL) methods are effective in continual offline RL (CORL) scenarios. In this study, we develop the Offline Continual World benchmark and demonstrate that traditional CL methods struggle with catastrophic forgetting, primarily due to the unique distribution shifts inherent to CORL scenarios. To address this challenge, we introduce CompoFormer, a structure-based continual transformer model that adaptively composes previous policies via a meta-policy network. Upon encountering a new task, CompoFormer leverages semantic correlations to selectively integrate relevant prior policies alongside newly trained parameters, thereby enhancing knowledge sharing and accelerating the learning process. Our experiments reveal that CompoFormer outperforms conventional CL methods, particularly in longer task sequences, showcasing a promising balance between plasticity and stability.

📄 PDF Abstract BibTeX arXiv:2411.11364

Code (1)

charleshsc/CompoFormer 공식 구현 pytorch

Tasks

Continual Learning Offline RL Reinforcement Learning (RL)

Denser $\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training

2026-07-02 · Meng Wang, Haohan Zhao, Wenzhuo Liu, Lu Yang 외 arxiv

Continual post-training enables foundation models to acquire new knowledge while preserving existing capabilities. Recent work suggests that on-policy learning can mitigate forgetting, with on-policy self-distillation em…

Reinforcement Learning Continual Learning

Continual-MAE: Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptation

2023-12-19 · CVPR 2024 1 · Jiaming Liu, ran Xu, Senqiao Yang, Renrui Zhang 외

Continual Test-Time Adaptation (CTTA) is proposed to migrate a source pre-trained model to continually changing target distributions, addressing real-world dynamism. Existing CTTA methods mainly rely on entropy minimizat…

Decoder Self-Supervised Learning Test-time Adaptation

Grow-Prune-Freeze Networks: Adaptive & Continual Learning Technique for Olfactory Navigation

2026-05-24 · Kordel K. France, Ovidiu Daescu arxiv

Training data for olfaction is scattered through disparate, non-standardized datasets that limit the ability to build representative world models. Olfactory navigation is a highly dynamic and non-stationary task that ben…

Reinforcement Learning Image Classification Continual Learning

Taming the Implicit: Dual-Channel Risk-Aware Reinforcement Fine-Tuning for Continual Multimodal Post-Training

2026-08-04 · Yibei Liu, Jiajun Chen, Qianle Zhang, Tangyue Jin 외 arxiv

Reinforcement fine-tuning (RFT) is widely believed to inherently resist catastrophic forgetting in continual post-training of multimodal large language models. Under pronounced task distributional shifts, however, forget…

Self-Distillation Enables Continual Learning

2026-01-27 · Idan Shenfeld, Mehul Damani, Jonas Hübotter, Pulkit Agrawal arxiv

Continual learning, enabling models to acquire new skills and knowledge without degrading existing capabilities, remains a fundamental challenge for foundation models. While on-policy reinforcement learning can reduce fo…