Federated Ensemble-Directed Offline Reinforcement Learning

2023-05-04 · Desik Rengarajan, Nitin Ragothaman, Dileep Kalathil, Srinivas Shakkottai

We consider the problem of federated offline reinforcement learning (RL), a scenario under which distributed learning agents must collaboratively learn a high-quality control policy only using small pre-collected datasets generated according to different unknown behavior policies. Na\"{i}vely combining a standard offline RL approach with a standard federated learning approach to solve this problem can lead to poorly performing policies. In response, we develop the Federated Ensemble-Directed Offline Reinforcement Learning Algorithm (FEDORA), which distills the collective wisdom of the clients using an ensemble learning approach. We develop the FEDORA codebase to utilize distributed compute resources on a federated learning platform. We show that FEDORA significantly outperforms other approaches, including offline RL over the combined data pool, in various complex continuous control environments and real-world datasets. Finally, we demonstrate the performance of FEDORA in the real-world on a mobile robot. We provide our code and a video of our experiments at \url{https://github.com/DesikRengarajan/FEDORA}.

📄 PDF Abstract BibTeX arXiv:2305.03097

Code (1)

desikrengarajan/fedora 공식 구현 pytorch

Tasks

continuous-control Continuous Control Ensemble Learning Federated Learning Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

FORLER: Federated Offline Reinforcement Learning with Q-Ensemble and Actor Rectification

2026-02-02 · Nan Qiao, Sheng Yue arxiv

In Internet-of-Things systems, federated learning has advanced online reinforcement learning (RL) by enabling parallel policy training without sharing raw data. However, interacting with real environments online can be r…

Reinforcement Learning Federated Learning Offline RL

Why so pessimistic? Estimating uncertainties for offline RL through ensembles, and why their independence matters.

2021-09-29 · Seyed Kamyar Seyed Ghasemipour, Shixiang Shane Gu, Ofir Nachum

In order to achieve strong performance in offline reinforcement learning (RL), it is necessary to act conservatively with respect to confident lower-bounds on anticipated values of actions. Thus, a valuable approach wou…

continuous-control Continuous Control D4RL Offline RL+2

Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters

2022-05-27 · Seyed Kamyar Seyed Ghasemipour, Shixiang Shane Gu, Ofir Nachum

Motivated by the success of ensembles for uncertainty estimation in supervised learning, we take a renewed look at how ensembles of $Q$-functions can be leveraged as the primary source of pessimism for offline reinforcem…

D4RL Offline RL Reinforcement Learning (RL)

FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Data

2025-12-02 · Nan Qiao, Sheng Yue, Ju Ren, Yaoxue Zhang arxiv

Offline Federated Reinforcement Learning (FRL), a marriage of federated learning and offline reinforcement learning, has attracted increasing interest recently. Albeit with some advancement, we find that the performance …

Reinforcement Learning Federated Learning

CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning

2022-04-18 · NAACL 2022 7 · Siddharth Verma, Justin Fu, Mengjiao Yang, Sergey Levine

Conventionally, generation of natural language for dialogue agents may be viewed as a statistical learning problem: determine the patterns in human-provided data and generate appropriate responses with similar statistica…

Chatbot Offline RL reinforcement-learning Reinforcement Learning+1