Harnessing Causality in Reinforcement Learning With Bagged Decision Times

2024-10-18 · Daiqi Gao, Hsin-Yu Lai, Predrag Klasnja, Susan A. Murphy

We consider reinforcement learning (RL) for a class of problems with bagged decision times. A bag contains a finite sequence of consecutive decision times. The transition dynamics are non-Markovian and non-stationary within a bag. All actions within a bag jointly impact a single reward, observed at the end of the bag. For example, in mobile health, multiple activity suggestions in a day collectively affect a user's daily commitment to being active. Our goal is to develop an online RL algorithm to maximize the discounted sum of the bag-specific rewards. To handle non-Markovian transitions within a bag, we utilize an expert-provided causal directed acyclic graph (DAG). Based on the DAG, we construct states as a dynamical Bayesian sufficient statistic of the observed history, which results in Markov state transitions within and across bags. We then formulate this problem as a periodic Markov decision process (MDP) that allows non-stationarity within a period. An online RL algorithm based on Bellman equations for stationary MDPs is generalized to handle periodic MDPs. We show that our constructed state achieves the maximal optimal value function among all state constructions for a periodic MDP. Finally, we evaluate the proposed method on testbed variants built from real data in a mobile health clinical trial.

📄 PDF Abstract BibTeX arXiv:2410.14659

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Reinforcement Learning from Bagged Reward

2024-02-06 · Yuting Tang, Xin-Qiang Cai, Yao-Xiang Ding, Qiyu Wu 외

In Reinforcement Learning (RL), it is commonly assumed that an immediate reward signal is generated for each action taken by the agent, helping the agent maximize cumulative rewards to obtain the optimal policy. However,…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A Survey on Causal Reinforcement Learning

2023-02-10 · Yan Zeng, Ruichu Cai, Fuchun Sun, Libo Huang 외

While Reinforcement Learning (RL) achieves tremendous success in sequential decision-making problems of many domains, it still faces key challenges of data inefficiency and the lack of interpretability. Interestingly, ma…

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+2

Causal Reinforcement Learning: A Survey

2023-07-04 · Zhihong Deng, Jing Jiang, Guodong Long, Chengqi Zhang

Reinforcement learning is an essential paradigm for solving sequential decision problems under uncertainty. Despite many remarkable achievements in recent decades, applying reinforcement learning methods in the real worl…

reinforcement-learning Reinforcement Learning Survey Transfer Learning

Bagged Boosted Trees for Classification of Ecological Momentary Assessment Data

2016-07-06 · Gerasimos Spanakis, Gerhard Weiss, Anne Roefs

Ecological Momentary Assessment (EMA) data is organized in multiple levels (per-subject, per-day, etc.) and this particular structure should be taken into account in machine learning algorithms used in EMA like decision …

BIG-bench Machine Learning General Classification

Object-Centric World Models for Causality-Aware Reinforcement Learning

2025-11-18 · Yosuke Nishimoto, Takashi Matsubara arxiv

World models have been developed to support sample-efficient deep reinforcement learning agents. However, it remains challenging for world models to accurately replicate environments that are high-dimensional, non-statio…

Reinforcement Learning