Efficient and Reliable Overlay Networks for Decentralized Federated Learning
We propose near-optimal overlay networks based on $d$-regular expander graphs to accelerate decentralized federated learning (DFL) and improve its generalization. In DFL a massive number of clients are connected by an overlay network, and they solve machine learning problems collaboratively without sharing raw data. Our overlay network design integrates spectral graph theory and the theoretical convergence and generalization bounds for DFL. As such, our proposed overlay networks accelerate convergence, improve generalization, and enhance robustness to clients failures in DFL with theoretical guarantees. Also, we present an efficient algorithm to convert a given graph to a practical overlay network and maintaining the network topology after potential client failures. We numerically verify the advantages of DFL with our proposed networks on various benchmark tasks, ranging from image classification to language modeling using hundreds of clients.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated LearningGeneralization Boundsimage-classificationImage ClassificationLanguage ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
PID-Guided Partial Alignment for Multimodal Decentralized Federated Learning
Multimodal decentralized federated learning (DFL) must support collaboration among agents that hold different modality subsets and often different model components, while operating over peer-to-peer (P2P) overlays withou…
Federated LearningOverlay-based Decentralized Federated Learning in Bandwidth-limited Networks
The emerging machine learning paradigm of decentralized federated learning (DFL) has the promise of greatly boosting the deployment of artificial intelligence (AI) by directly learning across distributed agents without c…
Federated LearningHEAL: Resilient and Self-* Hub-based Learning
Decentralized learning enhances privacy, scalability, and fault tolerance by distributing data and computation across nodes. A popular approach is Federated learning, which relies on a central aggregator, yet faces chall…
Federated LearningGenTorrent: Scaling Large Language Model Serving with An Overley Network
While significant progress has been made in research and development on open-source and cost-efficient large-language models (LLMs), serving scalability remains a critical challenge, particularly for small organizations …
Language ModelingLanguage ModellingLarge Language ModelSwiftBot: A Decentralized Platform for LLM-Powered Federated Robotic Task Execution
Federated robotic task execution systems require bridging natural language instructions to distributed robot control while efficiently managing computational resources across heterogeneous edge devices without centralize…