Decentralized and Model-Free Federated Learning: Consensus-Based Distillation in Function Space

2021-04-01 · Akihito Taya, Takayuki Nishio, Masahiro Morikura, Koji Yamamoto

This paper proposes a fully decentralized federated learning (FL) scheme for Internet of Everything (IoE) devices that are connected via multi-hop networks. Because FL algorithms hardly converge the parameters of machine learning (ML) models, this paper focuses on the convergence of ML models in function spaces. Considering that the representative loss functions of ML tasks e.g, mean squared error (MSE) and Kullback-Leibler (KL) divergence, are convex functionals, algorithms that directly update functions in function spaces could converge to the optimal solution. The key concept of this paper is to tailor a consensus-based optimization algorithm to work in the function space and achieve the global optimum in a distributed manner. This paper first analyzes the convergence of the proposed algorithm in a function space, which is referred to as a meta-algorithm, and shows that the spectral graph theory can be applied to the function space in a manner similar to that of numerical vectors. Then, consensus-based multi-hop federated distillation (CMFD) is developed for a neural network (NN) to implement the meta-algorithm. CMFD leverages knowledge distillation to realize function aggregation among adjacent devices without parameter averaging. An advantage of CMFD is that it works even with different NN models among the distributed learners. Although CMFD does not perfectly reflect the behavior of the meta-algorithm, the discussion of the meta-algorithm's convergence property promotes an intuitive understanding of CMFD, and simulation evaluations show that NN models converge using CMFD for several tasks. The simulation results also show that CMFD achieves higher accuracy than parameter aggregation for weakly connected networks, and CMFD is more stable than parameter aggregation methods.

📄 PDF Abstract BibTeX arXiv:2104.00352

Code (0)

등록된 구현이 없습니다.

Tasks

Federated Learning Knowledge Distillation

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

Convergence Visualizer of Decentralized Federated Distillation with Reduced Communication Costs

2023-12-19 · Akihito Taya, Yuuki Nishiyama, Kaoru Sezaki

Federated learning (FL) achieves collaborative learning without the need for data sharing, thus preventing privacy leakage. To extend FL into a fully decentralized algorithm, researchers have applied distributed optimiza…

Dimensionality Reduction Distributed Optimization Federated Learning

Function-Space ADMM for Decentralized Federated Learning: A Control Theoretic Perspective

2026-05-10 · Akihito Taya, Yuuki Nishiyama, Kaoru Sezaki arxiv

Decentralized federated learning (FL) is a promising approach for training machine learning models on sensor networks, Internet of Things (IoT) devices, and other edge systems where no central server exists. While federa…

Knowledge Distillation Federated Learning

Over-the-Air Decentralized Federated Learning

2021-06-15 · Yandong Shi, Yong Zhou, Yuanming Shi

In this paper, we consider decentralized federated learning (FL) over wireless networks, where over-the-air computation (AirComp) is adopted to facilitate the local model consensus in a device-to-device (D2D) communicati…

Federated Learning

Towards Serverless Semi-Decentralized Federated Learning with Heterogeneous Optimizers

2026-06-04 · Su Wang, Mung Chiang, H. Vincent Poor arxiv

We investigate cluster formation, involving the number and composition of clusters, in decentralized federated learning (FL) with heterogeneous machine learning (ML) optimizers. While clustering in centralized FL has ena…