Automatic Reward Shaping from Confounded Offline Data

2025-05-16 · Mingxuan Li, Junzhe Zhang, Elias Bareinboim

A key task in Artificial Intelligence is learning effective policies for controlling agents in unknown environments to optimize performance measures. Off-policy learning methods, like Q-learning, allow learners to make optimal decisions based on past experiences. This paper studies off-policy learning from biased data in complex and high-dimensional domains where \emph{unobserved confounding} cannot be ruled out a priori. Building on the well-celebrated Deep Q-Network (DQN), we propose a novel deep reinforcement learning algorithm robust to confounding biases in observed data. Specifically, our algorithm attempts to find a safe policy for the worst-case environment compatible with the observations. We apply our method to twelve confounded Atari games, and find that it consistently dominates the standard DQN in all games where the observed input to the behavioral and target policies mismatch and unobserved confounders exist.

📄 PDF Abstract BibTeX arXiv:2505.11478

Code (0)

등록된 구현이 없습니다.

Tasks

Atari GamesDeep Reinforcement LearningQ-Learning

Methods 이 논문이 사용한 방법론

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

Confounding Robust Continuous Control via Automatic Reward Shaping

2026-02-10 · Mateo Juliani, Mingxuan Li, Elias Bareinboim arxiv

Reward shaping has been applied widely to accelerate Reinforcement Learning (RL) agents' training. However, a principled way of designing effective reward shaping functions, especially for complex continuous control prob…

Reinforcement Learning Continuous Control

Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes

2022-09-18 · Zuyue Fu, Zhengling Qi, Zhaoran Wang, Zhuoran Yang 외

We study the offline reinforcement learning (RL) in the face of unmeasured confounders. Due to the lack of online interaction with the environment, offline RL is facing the following two significant challenges: (i) the a…

Offline RL reinforcement-learning Reinforcement Learning (RL)

STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order

2026-01-13 · Chengyang Gu, Yuxin Pan, Hui Xiong, Yize Chen arxiv

Offline reinforcement learning (RL) enables policy learning from pre-collected datasets, avoiding costly and risky online interactions, but it often struggles with long-horizon tasks involving sparse rewards. Existing go…

Reinforcement Learning Offline RL

ROLeR: Effective Reward Shaping in Offline Reinforcement Learning for Recommender Systems

2024-07-18 · Yi Zhang, Ruihong Qiu, Jiajun Liu, Sen Wang

Offline reinforcement learning (RL) is an effective tool for real-world recommender systems with its capacity to model the dynamic interest of users and its interactive nature. Most existing offline RL recommender system…

Offline RL Recommendation Systems reinforcement-learning Reinforcement Learning+1

Learning to Dynamically Select Between Reward Shaping Signals

2021-01-01 · Alexander Politowicz, Bing Liu

Reinforcement learning (RL) algorithms often have the limitation of sample complexity. Previous research has shown that the reliance on large amounts of experience can be mitigated through the presence of additional feed…

Reinforcement Learning (RL)