Decentralized Learning with Multi-Headed Distillation

2022-11-28 · CVPR 2023 1 · Andrey Zhmoginov, Mark Sandler, Nolan Miller, Gus Kristiansen, Max Vladymyrov

Decentralized learning with private data is a central problem in machine learning. We propose a novel distillation-based decentralized learning technique that allows multiple agents with private non-iid data to learn from each other, without having to share their data, weights or weight updates. Our approach is communication efficient, utilizes an unlabeled public dataset and uses multiple auxiliary heads for each client, greatly improving training efficiency in the case of heterogeneous data. This approach allows individual models to preserve and enhance performance on their private tasks while also dramatically improving their performance on the global aggregated data distribution. We study the effects of data and model architecture heterogeneity and the impact of the underlying communication graph topology on learning efficiency and show that our agents can significantly improve their performance compared to learning in isolation.

📄 PDF Abstract BibTeX arXiv:2211.15774

Code (0)

등록된 구현이 없습니다.

EnsembleNet: End-to-End Optimization of Multi-headed Models

2019-05-24 · Hanhan Li, Joe Yue-Hei Ng, Paul Natsev

Ensembling is a universally useful approach to boost the performance of machine learning models. However, individual models in an ensemble were traditionally trained independently in separate stages without information a…

Adaptive Distillation for Decentralized Learning from Heterogeneous Clients

2020-08-18 · Jiaxin Ma, Ryo Yonetani, Zahid Iqbal

This paper addresses the problem of decentralized learning to achieve a high-performance global model by asking a group of clients to share local models pre-trained with their own data resources. We are particularly inte…

Federated Learning

KD3A: Unsupervised Multi-Source Decentralized Domain Adaptation via Knowledge Distillation

2020-11-19 · Hao-Zhe Feng, Zhaoyang You, Minghao Chen, Tianye Zhang 외

Conventional unsupervised multi-source domain adaptation (UMDA) methods assume all source domains can be accessed directly. This neglects the privacy-preserving policy, that is, all the data and computations must be kept…

Domain Adaptation Knowledge Distillation Multi-Source Unsupervised Domain Adaptation Privacy Preserving+1

Deep Class Incremental Learning from Decentralized Data

2022-03-11 · Xiaohan Zhang, Songlin Dong, Jinjie Chen, Qi Tian 외

In this paper, we focus on a new and challenging decentralized machine learning paradigm in which there are continuous inflows of data to be addressed and the data are stored in multiple repositories. We initiate the stu…

class-incremental learning Class Incremental Learning Incremental Learning Knowledge Distillation

Towards Global Optimality in Cooperative MARL with the Transformation And Distillation Framework

2022-07-12 · Jianing Ye, Chenghao Li, Jianhao Wang, Chongjie Zhang

Decentralized execution is one core demand in cooperative multi-agent reinforcement learning (MARL). Recently, most popular MARL algorithms have adopted decentralized policies to enable decentralized execution and use gr…

Multi-agent Reinforcement Learning Policy Gradient Methods reinforcement-learning Reinforcement Learning (RL)