In-Context Compositional Q-Learning for Offline Reinforcement Learning

2025-09-28 · Qiushui Xu, Yuhao Huang, Yushu Jiang, Lei Song, Jinyu Wang, Wenliang Zheng, Jiang Bian arxiv

Accurate estimation of the Q-function is a central challenge in offline reinforcement learning. However, existing approaches often rely on a shared global Q-function, which is inadequate for capturing the compositional structure of tasks that consist of diverse subtasks. We propose In-context Compositional Q-Learning (ICQL), an offline RL framework that formulates Q-learning as a contextual inference problem and uses linear Transformers to adaptively infer local Q-functions from retrieved transitions without explicit subtask labels. Theoretically, we show that, under two assumptions -- linear approximability of the local Q-function and accurate inference of weights from retrieved context -- ICQL achieves a bounded approximation error for the Q-function and enables near-optimal policy extraction. Empirically, ICQL substantially improves performance in offline settings, achieving gains of up to 16.4% on kitchen tasks and up to 8.8% and 6.3% on MuJoCo and Adroit tasks, respectively. These results highlight the underexplored potential of in-context learning for robust and compositional value estimation and establish ICQL as a principled and effective framework for offline RL.

📄 PDF Abstract BibTeX arXiv:2509.24067

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning Offline RL

Compositional Transduction with Latent Analogies for Offline Goal-Conditioned Reinforcement Learning

2026-05-20 · Junseok Kim, Dohyeong Kim, Mineui Hong, Songhwai Oh arxiv

Compositional generalization is essential for reaching unseen goals under novel contextual variations in offline goal-conditioned reinforcement learning (GCRL), where a generalist goal-reaching agent must be learned from…

Reinforcement Learning

Robotic Manipulation Datasets for Offline Compositional Reinforcement Learning

2023-07-13 · Marcel Hussing, Jorge A. Mendez, Anisha Singrodia, Cassandra Kent 외

Offline reinforcement learning (RL) is a promising direction that allows RL agents to pre-train on large datasets, avoiding the recurrence of expensive data collection. To advance the field, it is crucial to generate lar…

Benchmarking Offline RL reinforcement-learning Reinforcement Learning+1

Compositional Conservatism: A Transductive Approach in Offline Reinforcement Learning

2024-04-06 · Yeda Song, Dongwook Lee, Gunhee Kim

Offline reinforcement learning (RL) is a compelling framework for learning optimal policies from past experiences without additional interaction with the environment. Nevertheless, offline RL inevitably faces the problem…

D4RL Offline RL reinforcement-learning Reinforcement Learning+1

A Policy-Guided Imitation Approach for Offline Reinforcement Learning

2022-10-15 · Haoran Xu, Li Jiang, Jianxiong Li, Xianyuan Zhan

Offline reinforcement learning (RL) methods can generally be categorized into two types: RL-based and Imitation-based. RL-based methods could in principle enjoy out-of-distribution generalization but suffer from erroneou…

D4RL Offline RL Off-policy evaluation Out-of-Distribution Generalization+3

Sim2O: Efficient Offline-to-Online MARL via Joint Action Composition

2026-06-19 · Bingchang Song, Yiqin Yang arxiv

Offline-to-online adaptation serves as a pivotal paradigm for mitigating the prohibitive cost of online exploration by bootstrapping reinforcement learning from offline datasets. While this paradigm has been extensively …

Multi-agent Reinforcement Learning