Pessimistic Causal Reinforcement Learning with Mediators for Confounded Offline Data

2024-03-18 · Danyang Wang, Chengchun Shi, Shikai Luo, Will Wei Sun

In real-world scenarios, datasets collected from randomized experiments are often constrained by size, due to limitations in time and budget. As a result, leveraging large observational datasets becomes a more attractive option for achieving high-quality policy learning. However, most existing offline reinforcement learning (RL) methods depend on two key assumptions--unconfoundedness and positivity--which frequently do not hold in observational data contexts. Recognizing these challenges, we propose a novel policy learning algorithm, PESsimistic CAusal Learning (PESCAL). We utilize the mediator variable based on front-door criterion to remove the confounding bias; additionally, we adopt the pessimistic principle to address the distributional shift between the action distributions induced by candidate policies, and the behavior policy that generates the observational data. Our key observation is that, by incorporating auxiliary variables that mediate the effect of actions on system dynamics, it is sufficient to learn a lower bound of the mediator distribution function, instead of the Q-function, to partially mitigate the issue of distributional shift. This insight significantly simplifies our algorithm, by circumventing the challenging task of sequential uncertainty quantification for the estimated Q-function. Moreover, we provide theoretical guarantees for the algorithms we propose, and demonstrate their efficacy through simulations, as well as real-world experiments utilizing offline datasets from a leading ride-hailing platform.

📄 PDF Abstract BibTeX arXiv:2403.11841

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Uncertainty Quantification

Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes

2022-05-26 · Miao Lu, Yifei Min, Zhaoran Wang, Zhuoran Yang

We study offline reinforcement learning (RL) in partially observable Markov decision processes. In particular, we aim to learn an optimal policy from a dataset collected by a behavior policy which possibly depends on the…

Causal Inference Offline RL Reinforcement Learning (RL)

Causal Flow Q-Learning for Robust Offline Reinforcement Learning

2026-02-02 · Mingxuan Li, Junzhe Zhang, Elias Bareinboim arxiv

Expressive policies based on flow-matching have been successfully applied in reinforcement learning (RL) more recently due to their ability to model complex action distributions from offline data. These algorithms build …

Reinforcement Learning Offline RL

Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning

2021-02-19 · Luofeng Liao, Zuyue Fu, Zhuoran Yang, Yixin Wang 외

In offline reinforcement learning (RL) an optimal policy is learned solely from a priori collected observational data. However, in observational data, actions are often confounded by unobserved variables. Instrumental va…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Provably Efficient Causal Reinforcement Learning with Confounded Observational Data

2020-06-22 · NeurIPS 2021 12 · Lingxiao Wang, Zhuoran Yang, Zhaoran Wang

Empowered by expressive function approximators such as neural networks, deep reinforcement learning (DRL) achieves tremendous empirical successes. However, learning expressive function approximators requires collecting a…

Autonomous Driving Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Pessimistic Auxiliary Policy for Offline Reinforcement Learning

2026-02-27 · Fan Zhang, Baoru Huang, Xin Zhang arxiv

Offline reinforcement learning aims to learn an agent from pre-collected datasets, avoiding unsafe and inefficient real-time interaction. However, inevitable access to out-ofdistribution actions during the learning proce…