Federated Reinforcement Learning with Constraint Heterogeneity

2024-05-06 · Hao Jin, Liangyu Zhang, Zhihua Zhang

We study a Federated Reinforcement Learning (FedRL) problem with constraint heterogeneity. In our setting, we aim to solve a reinforcement learning problem with multiple constraints while $N$ training agents are located in $N$ different environments with limited access to the constraint signals and they are expected to collaboratively learn a policy satisfying all constraint signals. Such learning problems are prevalent in scenarios of Large Language Model (LLM) fine-tuning and healthcare applications. To solve the problem, we propose federated primal-dual policy optimization methods based on traditional policy gradient methods. Specifically, we introduce $N$ local Lagrange functions for agents to perform local policy updates, and these agents are then scheduled to periodically communicate on their local policies. Taking natural policy gradient (NPG) and proximal policy optimization (PPO) as policy optimization methods, we mainly focus on two instances of our algorithms, ie, {FedNPG} and {FedPPO}. We show that FedNPG achieves global convergence with an $\tilde{O}(1/\sqrt{T})$ rate, and FedPPO efficiently solves complicated learning tasks with the use of deep neural networks.

📄 PDF Abstract BibTeX arXiv:2405.03236

Code (0)

등록된 구현이 없습니다.

Tasks

Language ModelingLanguage ModellingLarge Language ModelPolicy Gradient Methodsreinforcement-learning Reinforcement Learning

Methods 이 논문이 사용한 방법론

Focus 설명 없음

Federated Reinforcement Learning with Environment Heterogeneity

2022-04-06 · Hao Jin, Yang Peng, Wenhao Yang, Shusen Wang 외

We study a Federated Reinforcement Learning (FedRL) problem in which $n$ agents collaboratively learn a single policy without sharing the trajectories they collected during agent-environment interaction. We stress the co…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

FedMRL: Data Heterogeneity Aware Federated Multi-agent Deep Reinforcement Learning for Medical Imaging

2024-07-08 · Pranab Sahoo, Ashutosh Tripathi, Sriparna Saha, Samrat Mondal

Despite recent advancements in federated learning (FL) for medical image diagnosis, addressing data heterogeneity among clients remains a significant challenge for practical implementation. A primary hurdle in FL arises …

Deep Reinforcement Learning Fairness Federated Learning Multi-agent Reinforcement Learning+2

Federated Stochastic Approximation under Markov Noise and Heterogeneity: Applications in Reinforcement Learning

2022-06-21 · Sajad Khodadadian, Pranay Sharma, Gauri Joshi, Siva Theja Maguluri

Since reinforcement learning algorithms are notoriously data-intensive, the task of sampling observations from the environment is usually split across multiple agents. However, transferring these observations from the ag…

Q-Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Adaptive Federated Learning and Digital Twin for Industrial Internet of Things

2020-10-25 · Wen Sun, Shiyu Lei, Lu Wang, Zhiqiang Liu 외

Industrial Internet of Things (IoT) enables distributed intelligent services varying with the dynamic and realtime industrial devices to achieve Industry 4.0 benefits. In this paper, we consider a new architecture of dig…

Clustering Deep Reinforcement Learning Federated Learning Reinforcement Learning (RL)

Navigating High-Degree Heterogeneity: Federated Learning in Aerial and Space Networks

2024-06-25 · Fan Dong, Henry Leung, Steve Drew

Federated learning offers a compelling solution to the challenges of networking and data privacy within aerial and space networks by utilizing vast private edge data and computing capabilities accessible through drones, …

Federated Learning