Federated Offline Reinforcement Learning

2022-06-11 · Doudou Zhou, Yufeng Zhang, Aaron Sonabend-W, Zhaoran Wang, Junwei Lu, Tianxi Cai

Evidence-based or data-driven dynamic treatment regimes are essential for personalized medicine, which can benefit from offline reinforcement learning (RL). Although massive healthcare data are available across medical institutions, they are prohibited from sharing due to privacy constraints. Besides, heterogeneity exists in different sites. As a result, federated offline RL algorithms are necessary and promising to deal with the problems. In this paper, we propose a multi-site Markov decision process model that allows for both homogeneous and heterogeneous effects across sites. The proposed model makes the analysis of the site-level features possible. We design the first federated policy optimization algorithm for offline RL with sample complexity. The proposed algorithm is communication-efficient, which requires only a single round of communication interaction by exchanging summary statistics. We give a theoretical guarantee for the proposed algorithm, where the suboptimality for the learned policies is comparable to the rate as if data is not distributed. Extensive simulations demonstrate the effectiveness of the proposed algorithm. The method is applied to a sepsis dataset in multiple sites to illustrate its use in clinical settings.

📄 PDF Abstract BibTeX arXiv:2206.05581

Code (0)

등록된 구현이 없습니다.

Tasks

Offline RLPrivacy Preservingreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Federated Ensemble-Directed Offline Reinforcement Learning

2023-05-04 · Desik Rengarajan, Nitin Ragothaman, Dileep Kalathil, Srinivas Shakkottai

We consider the problem of federated offline reinforcement learning (RL), a scenario under which distributed learning agents must collaboratively learn a high-quality control policy only using small pre-collected dataset…

continuous-control Continuous Control Ensemble Learning Federated Learning+4

FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Data

2025-12-02 · Nan Qiao, Sheng Yue, Ju Ren, Yaoxue Zhang arxiv

Offline Federated Reinforcement Learning (FRL), a marriage of federated learning and offline reinforcement learning, has attracted increasing interest recently. Albeit with some advancement, we find that the performance …

Reinforcement Learning Federated Learning

Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices

2024-02-08 · Jiin Woo, Laixi Shi, Gauri Joshi, Yuejie Chi

Offline reinforcement learning (RL), which seeks to learn an optimal policy using offline data, has garnered significant interest due to its potential in critical applications where online data collection is infeasible o…

Federated Learning Offline RL Q-Learning reinforcement-learning+2

FORLER: Federated Offline Reinforcement Learning with Q-Ensemble and Actor Rectification

2026-02-02 · Nan Qiao, Sheng Yue arxiv

In Internet-of-Things systems, federated learning has advanced online reinforcement learning (RL) by enabling parallel policy training without sharing raw data. However, interacting with real environments online can be r…

Reinforcement Learning Federated Learning Offline RL

Federated Offline Policy Optimization with Dual Regularization

2024-05-24 · Sheng Yue, Zerui Qin, Xingyuan Hua, Yongheng Deng 외

Federated Reinforcement Learning (FRL) has been deemed as a promising solution for intelligent decision-making in the era of Artificial Internet of Things. However, existing FRL approaches often entail repeated interacti…

Decision Making