홈 › Papers

Yes, Q-learning Helps Offline In-Context RL

2025-02-24 · Denis Tarasov, Alexander Nikulin, Ilya Zisman, Albina Klepach, Andrei Polubarov, Nikita Lyubaykin, Alexander Derevyagin, Igor Kiselev, Vladislav Kurenkov

Existing offline in-context reinforcement learning (ICRL) methods have predominantly relied on supervised training objectives, which are known to have limitations in offline RL settings. In this study, we explore the integration of RL objectives within an offline ICRL framework. Through experiments on more than 150 GridWorld and MuJoCo environment-derived datasets, we demonstrate that optimizing RL objectives directly improves performance by approximately 30% on average compared to widely adopted Algorithm Distillation (AD), across various dataset coverages, structures, expertise levels, and environmental complexities. Furthermore, in the challenging XLand-MiniGrid environment, RL objectives doubled the performance of AD. Our results also reveal that the addition of conservatism during value learning brings additional improvements in almost all settings tested. Our findings emphasize the importance of aligning ICRL learning objectives with the RL reward-maximization goal, and demonstrate that offline RL is a promising direction for advancing ICRL.

📄 PDF Abstract BibTeX arXiv:2502.17666

Code (0)

등록된 구현이 없습니다.

Tasks

In-Context Reinforcement LearningMuJoCoOffline RLQ-LearningReinforcement Learning (RL)

Offline A/B testing for Recommender Systems

2018-01-22 · Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham 외

Before A/B testing online a new version of a recommender system, it is usual to perform some offline evaluations on historical data. We focus on evaluation methods that compute an estimator of the potential uplift in rev…

counterfactual Product Recommendation Recommendation Systems

Masking Stale Observations Helps Search Agents -- Until It Doesn't: A Regime Map and Its Mechanism

2026-05-29 · Haoxiang Zhang, Qixin Xu, Zhuofeng Li, Lei Zhang 외 arxiv

Long-horizon search agents accumulate large amounts of retrieved content across many tool calls, making context-budget efficiency increasingly important. A minimal intervention is to mask stale observations from the cont…

Large-Scale Retrieval for Reinforcement Learning

2022-06-10 · Peter C. Humphreys, Arthur Guez, Olivier Tieleman, Laurent SIfre 외

Effective decision making involves flexibly relating past experiences and relevant contextual information to a novel situation. In deep reinforcement learning (RL), the dominant paradigm is for an agent to amortise infor…

Decision Making Deep Reinforcement Learning Offline RL reinforcement-learning+3

Stay Within Your Bounds: Distance-Guided Decoding for Guaranteed Context-Free Grammar Compliance

2026-08-28 · Vincenzo Collura, Karim Tit, Eleonora Giunchiglia, Mike Papadakis 외 arxiv

Grammar-constrained decoding helps large language models produce syntactically valid structured outputs, such as code, JSON, and SQL. For context-free grammars, many practical decoders enforce local prefix feasibility: e…

COOPO: Cyclic Offline-Online Policy Optimization Algorithm

2026-05-18 · Qisai Liu, Zhanhong Jiang, Joshua Russell Waite, Aditya Balu 외 arxiv

Offline reinforcement learning struggles with distributional shift and constrained performance due to static dataset limitations, while online RL demands prohibitive environment interactions. The recent advent of hybrid …

Reinforcement Learning