Active Measuring in Reinforcement Learning With Delayed Negative Effects

2025-10-16 · Daiqi Gao, Ziping Xu, Aseel Rawashdeh, Predrag Klasnja, Susan A. Murphy arxiv

Measuring states in reinforcement learning (RL) can be costly in real-world settings and may negatively influence future outcomes. We introduce the Actively Observable Markov Decision Process (AOMDP), where an agent not only selects control actions but also decides whether to measure the latent state. The measurement action reveals the true latent state but may have a negative delayed effect on the environment. We show that this reduced uncertainty may provably improve sample efficiency and increase the value of the optimal policy despite these costs. We formulate an AOMDP as a periodic partially observable MDP and propose an online RL algorithm based on belief states. To approximate the belief states, we further propose a sequential Monte Carlo method to jointly approximate the posterior of unknown static environment parameters and unobserved latent states. We evaluate the proposed algorithm in a digital health application, where the agent decides when to deliver digital interventions and when to assess users' health status through surveys.

📄 PDF Abstract BibTeX arXiv:2510.14315

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

A Nonparametric Delayed Feedback Model for Conversion Rate Prediction

2018-02-01 · Yuya Yoshikawa, Yusaku Imai

Predicting conversion rates (CVRs) in display advertising (e.g., predicting the proportion of users who purchase an item (i.e., a conversion) after its corresponding ad is clicked) is important when measuring the effects…

Prediction

Delay-Empowered Causal Hierarchical Reinforcement Learning

2026-05-12 · Chenran Zhao, Dianxi Shi, Haotian Wang, Mengzhu Wang 외 arxiv

Many real-world tasks involve delayed effects, where the outcomes of actions emerge after varying time lags. Existing delay-aware reinforcement learning methods often rely on state augmentation, prior knowledge of delay …

Hierarchical Reinforcement Learning

Learning Personalized Ad Impact via Contextual Reinforcement Learning under Delayed Rewards

2025-10-22 · Yuwei Cheng, Zifeng Zhao, Haifeng Xu arxiv

Online advertising platforms use automated auctions to connect advertisers with potential customers, requiring effective bidding strategies to maximize profits. Accurate ad impact estimation requires considering three ke…

Reinforcement Learning

Reinforcement Learning from Delayed Observations via World Models

2024-03-18 · Armin Karamzade, KyungMin Kim, Montek Kalsi, Roy Fox

In standard reinforcement learning settings, agents typically assume immediate feedback about the effects of their actions after taking them. However, in practice, this assumption may not hold true due to physical constr…

continuous-control Continuous Control reinforcement-learning Reinforcement Learning

A Reinforcement Learning Framework for Dynamic Mediation Analysis

2023-01-31 · Lin Ge, Jitao Wang, Chengchun Shi, Zhenke Wu 외

Mediation analysis learns the causal effect transmitted via mediator variables between treatments and outcomes and receives increasing attention in various scientific domains to elucidate causal relations. Most existing …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)