Convergence Visualizer of Decentralized Federated Distillation with Reduced Communication Costs

2023-12-19 · Akihito Taya, Yuuki Nishiyama, Kaoru Sezaki

Federated learning (FL) achieves collaborative learning without the need for data sharing, thus preventing privacy leakage. To extend FL into a fully decentralized algorithm, researchers have applied distributed optimization algorithms to FL by considering machine learning (ML) tasks as parameter optimization problems. Conversely, the consensus-based multi-hop federated distillation (CMFD) proposed in the authors' previous work makes neural network (NN) models get close with others in a function space rather than in a parameter space. Hence, this study solves two unresolved challenges of CMFD: (1) communication cost reduction and (2) visualization of model convergence. Based on a proposed dynamic communication cost reduction method (DCCR), the amount of data transferred in a network is reduced; however, with a slight degradation in the prediction accuracy. In addition, a technique for visualizing the distance between the NN models in a function space is also proposed. The technique applies a dimensionality reduction technique by approximating infinite-dimensional functions as numerical vectors to visualize the trajectory of how the models change by the distributed learning algorithm.

📄 PDF Abstract BibTeX arXiv:2312.11905

Code (0)

등록된 구현이 없습니다.

Tasks

Dimensionality Reduction Distributed Optimization Federated Learning

Personalized Decentralized Federated Learning with Knowledge Distillation

2023-02-23 · Eunjeong Jeong, Marios Kountouris

Personalization in federated learning (FL) functions as a coordinator for clients with high variance in data or behavior. Ensuring the convergence of these clients' models relies on how closely users collaborate with tho…

Federated Learning Knowledge Distillation

Boosting the Performance of Decentralized Federated Learning via Catalyst Acceleration

2024-10-09 · Qinglun Li, Miao Zhang, Yingqi Liu, Quanjun Yin 외

Decentralized Federated Learning has emerged as an alternative to centralized architectures due to its faster training, privacy preservation, and reduced communication overhead. In decentralized communication, the server…

Federated Learning

Function-Space ADMM for Decentralized Federated Learning: A Control Theoretic Perspective

2026-05-10 · Akihito Taya, Yuuki Nishiyama, Kaoru Sezaki arxiv

Decentralized federated learning (FL) is a promising approach for training machine learning models on sensor networks, Internet of Things (IoT) devices, and other edge systems where no central server exists. While federa…

Knowledge Distillation Federated Learning

Decentralized Federated Learning: A Segmented Gossip Approach

2019-08-21 · Chenghao Hu, Jingyan Jiang, Zhi Wang

The emerging concern about data privacy and security has motivated the proposal of federated learning, which allows nodes to only synchronize the locally-trained models instead their own original data. Conventional feder…