Causal Deep Reinforcement Learning Using Observational Data

2022-11-28 · Wenxuan Zhu, Chao Yu, Qiang Zhang

Deep reinforcement learning (DRL) requires the collection of interventional data, which is sometimes expensive and even unethical in the real world, such as in the autonomous driving and the medical field. Offline reinforcement learning promises to alleviate this issue by exploiting the vast amount of observational data available in the real world. However, observational data may mislead the learning agent to undesirable outcomes if the behavior policy that generates the data depends on unobserved random variables (i.e., confounders). In this paper, we propose two deconfounding methods in DRL to address this problem. The methods first calculate the importance degree of different samples based on the causal inference technique, and then adjust the impact of different samples on the loss function by reweighting or resampling the offline dataset to ensure its unbiasedness. These deconfounding methods can be flexibly combined with existing model-free DRL algorithms such as soft actor-critic and deep Q-learning, provided that a weak condition can be satisfied by the loss functions of these algorithms. We prove the effectiveness of our deconfounding methods and validate them experimentally.

📄 PDF Abstract BibTeX arXiv:2211.15355

Code (0)

등록된 구현이 없습니다.

Tasks

Autonomous Driving Causal InferenceDeep Reinforcement LearningQ-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbacks

2026-03-19 · Hao Wang, Licheng Pan, Zhichao Chen, Chunyuan Zheng 외 arxiv

Despite the success of reinforcement learning from human feedback (RLHF) in aligning language models, current reward modeling heavily relies on experimental feedback data collected from human annotators under controlled …

Reinforcement Learning

Causal Reinforcement Learning using Observational and Interventional Data

2021-06-28 · Maxime Gasse, Damien Grasset, Guillaume Gaudron, Pierre-Yves Oudeyer

Learning efficiently a causal model of the environment is a key challenge of model-based RL agents operating in POMDPs. We consider here a scenario where the learning agent has the ability to collect online experiences t…

Causal Inference Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Training a Resilient Q-Network against Observational Interference

2021-02-18 · Chao-Han Huck Yang, I-Te Danny Hung, Yi Ouyang, Pin-Yu Chen

Deep reinforcement learning (DRL) has demonstrated impressive performance in various gaming simulators and real-world applications. In practice, however, a DRL agent may receive faulty observation by abrupt interferences…

Causal Inference Deep Reinforcement Learning

Provably Efficient Causal Reinforcement Learning with Confounded Observational Data

2020-06-22 · NeurIPS 2021 12 · Lingxiao Wang, Zhuoran Yang, Zhaoran Wang

Empowered by expressive function approximators such as neural networks, deep reinforcement learning (DRL) achieves tremendous empirical successes. However, learning expressive function approximators requires collecting a…

Autonomous Driving Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+1

Causal Reasoning from Meta-reinforcement Learning

2019-01-23 · ICLR 2019 5 · Ishita Dasgupta, Jane Wang, Silvia Chiappa, Jovana Mitrovic 외

Discovering and exploiting the causal structure in the environment is a crucial challenge for intelligent agents. Here we explore whether causal reasoning can emerge via meta-reinforcement learning. We train a recurrent …

counterfactual Meta Reinforcement Learning reinforcement-learning Reinforcement Learning+1