FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Data

2025-12-02 · Nan Qiao, Sheng Yue, Ju Ren, Yaoxue Zhang arxiv

Offline Federated Reinforcement Learning (FRL), a marriage of federated learning and offline reinforcement learning, has attracted increasing interest recently. Albeit with some advancement, we find that the performance of most existing offline FRL methods drops dramatically when provided with mixed-quality data, that is, the logging behaviors (offline data) are collected by policies with varying qualities across clients. To overcome this limitation, this paper introduces a new vote-based offline FRL framework, named FOVA. It exploits a \emph{vote mechanism} to identify high-return actions during local policy evaluation, alleviating the negative effect of low-quality behaviors from diverse local learning policies. Besides, building on advantage-weighted regression (AWR), we construct consistent local and global training objectives, significantly enhancing the efficiency and stability of FOVA. Further, we conduct an extensive theoretical analysis and rigorously show that the policy learned by FOVA enjoys strict policy improvement over the behavioral policy. Extensive experiments corroborate the significant performance gains of our proposed algorithm over existing baselines on widely used benchmarks.

📄 PDF Abstract BibTeX arXiv:2512.02350

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning Federated Learning

Federated Ensemble-Directed Offline Reinforcement Learning

2023-05-04 · Desik Rengarajan, Nitin Ragothaman, Dileep Kalathil, Srinivas Shakkottai

We consider the problem of federated offline reinforcement learning (RL), a scenario under which distributed learning agents must collaboratively learn a high-quality control policy only using small pre-collected dataset…

continuous-control Continuous Control Ensemble Learning Federated Learning+4

FORLER: Federated Offline Reinforcement Learning with Q-Ensemble and Actor Rectification

2026-02-02 · Nan Qiao, Sheng Yue arxiv

In Internet-of-Things systems, federated learning has advanced online reinforcement learning (RL) by enabling parallel policy training without sharing raw data. However, interacting with real environments online can be r…

Reinforcement Learning Federated Learning Offline RL

Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices

2024-02-08 · Jiin Woo, Laixi Shi, Gauri Joshi, Yuejie Chi

Offline reinforcement learning (RL), which seeks to learn an optimal policy using offline data, has garnered significant interest due to its potential in critical applications where online data collection is infeasible o…

Federated Learning Offline RL Q-Learning reinforcement-learning+2

A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective

2024-03-12 · Yunpeng Qing, Shunyu Liu, Jingyuan Cong, KaiXuan Chen 외

Offline reinforcement learning endeavors to leverage offline datasets to craft effective agent policy without online interaction, which imposes proper conservative constraints with the support of behavior policies to tac…

D4RL reinforcement-learning Reinforcement Learning (RL)

Improving Behavioural Cloning with Positive Unlabeled Learning

2023-01-27 · Qiang Wang, Robert McCarthy, David Cordova Bulens, Kevin McGuinness 외

Learning control policies offline from pre-recorded datasets is a promising avenue for solving challenging real-world problems. However, available datasets are typically of mixed quality, with a limited number of the tra…

Behavioural cloning D4RL Imitation Learning reinforcement-learning+1