Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applications
There is increasing interest in data-driven approaches for recommending optimal treatment strategies in many chronic disease management and critical care applications. Reinforcement learning methods are well-suited to this sequential decision-making problem, but must be trained and evaluated exclusively on retrospective medical record datasets as direct online exploration is unsafe and infeasible. Despite this requirement, the vast majority of treatment optimization studies use off-policy RL methods (e.g., Double Deep Q Networks (DDQN) or its variants) that are known to perform poorly in purely offline settings. Recent advances in offline RL, such as Conservative Q-Learning (CQL), offer a suitable alternative. But there remain challenges in adapting these approaches to real-world applications where suboptimal examples dominate the retrospective dataset and strict safety constraints need to be satisfied. In this work, we introduce a practical and theoretically grounded transition sampling approach to address action imbalance during offline RL training. We perform extensive experiments on two real-world tasks for diabetes and sepsis treatment optimization to compare performance of the proposed approach against prominent off-policy and offline RL baselines (DDQN and CQL). Across a range of principled and clinically relevant metrics, we show that our proposed approach enables substantial improvements in expected health outcomes and in accordance with relevant practice and safety guidelines.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingManagementOffline RLQ-Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Sequential Decision MakingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Fat-to-Thin Policy Optimization: Offline RL with Sparse Policies
Sparse continuous policies are distributions that can choose some actions at random yet keep strictly zero probability for the other actions, which are radically different from the Gaussian. They have important real-worl…
MuJoCoOffline RLLearning Optimal Treatment Strategies for Sepsis Using Offline Reinforcement Learning in Continuous Space
Sepsis is a leading cause of death in the ICU. It is a disease requiring complex interventions in a short period of time, but its optimal treatment strategy remains uncertain. Evidence suggests that the practices of curr…
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Selective Uncertainty Propagation in Offline RL
We consider the finite-horizon offline reinforcement learning (RL) setting, and are motivated by the challenge of learning the policy at any step h in dynamic programming (DP) algorithms. To learn this, it is sufficient …
Offline RLreinforcement-learningReinforcement Learning (RL)Data Poisoning Attacks in Contextual Bandits
We study offline data poisoning attacks in contextual bandits, a class of reinforcement learning problems with important applications in online recommendation and adaptive medical treatment, among others. We provide a ge…
Data PoisoningMulti-Armed Banditsreinforcement-learningReinforcement Learning+1Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategies
When applying offline reinforcement learning (RL) in healthcare scenarios, the out-of-distribution (OOD) issues pose significant risks, as inappropriate generalization beyond clinical expertise can result in potentially …
Offline RLQ-Learningreinforcement-learningReinforcement Learning+2