Global Convergence Guarantees for Federated Policy Gradient Methods with Adversaries
Federated Reinforcement Learning (FRL) allows multiple agents to collaboratively build a decision making policy without sharing raw trajectories. However, if a small fraction of these agents are adversarial, it can lead to catastrophic results. We propose a policy gradient based approach that is robust to adversarial agents which can send arbitrary values to the server. Under this setting, our results form the first global convergence guarantees with general parametrization. These results demonstrate resilience with adversaries, while achieving optimal sample complexity of order $\tilde{\mathcal{O}}\left( \frac{1}{N\epsilon^2} \left( 1+ \frac{f^2}{N}\right)\right)$, where $N$ is the total number of agents and $f<N/2$ is the number of adversarial agents.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingPolicy Gradient MethodsSimilar Papers 제목 키워드 기반
Federated Natural Policy Gradient and Actor Critic Methods for Multi-task Reinforcement Learning
Federated reinforcement learning (RL) enables collaborative decision making of multiple distributed agents without sharing local data trajectories. In this work, we consider a multi-task setting, in which each agent has …
Decision MakingPolicy Gradient Methodsreinforcement-learningReinforcement Learning (RL)Towards Fast Rates for Federated and Multi-Task Reinforcement Learning
We consider a setting involving $N$ agents, where each agent interacts with an environment modeled as a Markov Decision Process (MDP). The agents' MDPs differ in their reward functions, capturing heterogeneous objectives…
reinforcement-learningReinforcement LearningOn Global Convergence Rates for Federated Policy Gradient under Heterogeneous Environment
Ensuring convergence of policy gradient methods in federated reinforcement learning (FRL) under environment heterogeneity remains a major challenge. In this work, we first establish that heterogeneity, perhaps counter-in…
Federated LearningPolicy Gradient MethodsQ-LearningGlobal Convergence of Policy Gradient Methods in Reinforcement Learning, Games and Control
Policy gradient methods, where one searches for the policy of interest by maximizing the value functions using first-order information, become increasingly popular for sequential decision making in reinforcement learning…
Decision MakingPolicy Gradient Methodsreinforcement-learningSequential Decision MakingImproved Communication Efficiency in Federated Natural Policy Gradient via ADMM-based Gradient Updates
Federated reinforcement learning (FedRL) enables agents to collaboratively train a global policy without sharing their individual data. However, high communication overhead remains a critical bottleneck, particularly for…
MuJoCo