Counterfactual Transport Flows for Offline Conservative Trajectory Refinement

2026-06-08 · Lena Krieger, Xuan Zhao, Zhuo Cao, Qin Wang, Hanno Scharr, Ira Assent arxiv

Offline reinforcement learning (RL) offers a path to policy improvement from logged data alone, using historical returns or other measurable outcomes as world feedback. A key difficulty is improving observed behavior without extrapolating beyond what the offline data supports. We propose \emph{counterfactual transport flows}, a source-conditioned trajectory refinement framework for offline decision-making guided by world feedback. Given a low-feedback candidate trajectory, we construct local preference pairs from offline data by retrieving nearby trajectories in latent trajectory space with higher task-specific feedback, and use them as weak supervision for conservative refinement. The framework learns instance-specific refinement directions: at inference time, a refinement strength parameter controls how far the candidate trajectory is transported, enabling a trade-off between preserving the original behavior and applying stronger improvement. Experiments on D4RL benchmarks, including AntMaze and MuJoCo tasks, show that our method improves behavior from historical returns as world feedback, while providing interpretable trajectory-level refinement paths.

📄 PDF Abstract BibTeX arXiv:2606.09115

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning

2023-09-22 · NeurIPS 2023 11 · Jianzhun Shao, Yun Qu, Chen Chen, Hongchang Zhang 외

Offline multi-agent reinforcement learning is challenging due to the coupling effect of both distribution shift issue common in offline setting and the high dimension issue common in multi-agent setting, making the actio…

counterfactual Multi-agent Reinforcement Learning Offline RL Q-Learning+2

Counterfactual Identifiability via Dynamic Optimal Transport

2025-10-09 · Fabio De Sousa Ribeiro, Ainkaran Santhirasekaram, Ben Glocker arxiv

We address the open question of counterfactual identification for high-dimensional multivariate outcomes from observational data. Pearl (2000) argues that counterfactuals must be identifiable (i.e., recoverable from the …

Let Offline RL Flow: Training Conservative Agents in the Latent Space of Normalizing Flows

2022-11-20 · Dmitriy Akimov, Vladislav Kurenkov, Alexander Nikulin, Denis Tarasov 외

Offline reinforcement learning aims to train a policy on a pre-recorded and fixed dataset without any additional environment interactions. There are two major challenges in this setting: (1) extrapolation error caused by…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Coordinating Ride-Pooling with Public Transit using Reward-Guided Conservative Q-Learning: An Offline Training and Online Fine-Tuning Reinforcement Learning Framework

2025-01-24 · Yulong Hu, Tingting Dong, Sen Li

This paper introduces a novel reinforcement learning (RL) framework, termed Reward-Guided Conservative Q-learning (RG-CQL), to enhance coordination between ride-pooling and public transit within a multimodal transportati…

Q-Learning Reinforcement Learning (RL)

Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learning

2024-02-13 · Eslam Eldeeb, Houssem Sifaou, Osvaldo Simeone, Mohammad Shehab 외

Reinforcement learning (RL) has been widely adopted for controlling and optimizing complex engineering systems such as next-generation wireless networks. An important challenge in adopting RL is the need for direct acces…

Multi-agent Reinforcement Learning Q-Learning quantile regression reinforcement-learning+3