Proxy Experience Replay: Federated Distillation for Distributed Reinforcement Learning
Traditional distributed deep reinforcement learning (RL) commonly relies on exchanging the experience replay memory (RM) of each agent. Since the RM contains all state observations and action policy history, it may incur huge communication overhead while violating the privacy of each agent. Alternatively, this article presents a communication-efficient and privacy-preserving distributed RL framework, coined federated reinforcement distillation (FRD). In FRD, each agent exchanges its proxy experience replay memory (ProxRM), in which policies are locally averaged with respect to proxy states clustering actual states. To provide FRD design insights, we present ablation studies on the impact of ProxRM structures, neural network architectures, and communication intervals. Furthermore, we propose an improved version of FRD, coined mixup augmented FRD (MixFRD), in which ProxRM is interpolated using the mixup data augmentation algorithm. Simulations in a Cartpole environment validate the effectiveness of MixFRD in reducing the variance of mission completion time and communication cost, compared to the benchmark schemes, vanilla FRD, federated reinforcement learning (FRL), and policy distillation (PD).
Code (0)
등록된 구현이 없습니다.
Tasks
ClusteringData AugmentationDeep Reinforcement LearningPrivacy Preservingreinforcement-learningReinforcement LearningReinforcement Learning (RL)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Federated Reinforcement Distillation with Proxy Experience Memory
In distributed reinforcement learning, it is common to exchange the experience memory of each agent and thereby collectively train their local models. The experience memory, however, contains all the preceding state obse…
Privacy Preservingreinforcement-learningReinforcement LearningReinforcement Learning (RL)Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learning
Federated Class Continual Learning (FCCL) merges the challenges of distributed client learning with the need for seamless adaptation to new classes without forgetting old ones. The key challenge in FCCL is catastrophic f…
Continual LearningContrastive LearningData-free Knowledge DistillationDomain Generalization+1FlashbackCL: Mitigating Temporal Forgetting in Federated Learning
Federated Learning (FL) of foundation and edge models increasingly targets deployments where client data distributions drift over time, yet existing forgetting-mitigation methods assume each client's distribution is stat…
Federated LearningContinual LearningFedDTG:Federated Data-Free Knowledge Distillation via Three-Player Generative Adversarial Networks
While existing federated learning approaches primarily focus on aggregating local models to construct a global model, in realistic settings, some clients may be reluctant to share their private models due to the inclusio…
Data-free Knowledge DistillationFederated LearningGenerative Adversarial NetworkKnowledge DistillationExploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillation
Federated learning (FL) is a privacy-preserving machine learning paradigm in which the server periodically aggregates local model parameters from clients without assembling their private data. Constrained communication a…
Federated LearningPrivacy Preserving