Decentralized Learning Made Easy with DecentralizePy
Decentralized learning (DL) has gained prominence for its potential benefits in terms of scalability, privacy, and fault tolerance. It consists of many nodes that coordinate without a central server and exchange millions of parameters in the inherently iterative process of machine learning (ML) training. In addition, these nodes are connected in complex and potentially dynamic topologies. Assessing the intricate dynamics of such networks is clearly not an easy task. Often in literature, researchers resort to simulated environments that do not scale and fail to capture practical and crucial behaviors, including the ones associated to parallelism, data transfer, network delays, and wall-clock time. In this paper, we propose DecentralizePy, a distributed framework for decentralized ML, which allows for the emulation of large-scale learning networks in arbitrary topologies. We demonstrate the capabilities of DecentralizePy by deploying techniques such as sparsification and secure aggregation on top of several topologies, including dynamic networks with more than one thousand nodes.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A further exploration of deep Multi-Agent Reinforcement Learning with Hybrid Action Space
The research of extending deep reinforcement learning (drl) to multi-agent field has solved many complicated problems and made great achievements. However, almost all these studies only focus on discrete or continuous ac…
Deep Reinforcement LearningMulti-agent Reinforcement Learningreinforcement-learningReinforcement Learning+1Decentralized Finance (DeFi)
Decentralized finance, powered by blockchain technology, is growing day by day. This field, which emerged a few years ago, today manages $70 billion in assets. In this study, the concept of decentralized finance is discu…
BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep Learning
Decentralized algorithm is a form of computation that achieves a global goal through local dynamics that relies on low-cost communication between directly-connected agents. On large-scale optimization tasks involving dis…
Deep LearningFed-DART and FACT: A solution for Federated Learning in a production environment
Federated Learning as a decentralized artificial intelligence (AI) solution solves a variety of problems in industrial applications. It enables a continuously self-improving AI, which can be deployed everywhere at the ed…
Federated LearningAdjacent Leader Decentralized Stochastic Gradient Descent
This work focuses on the decentralized deep learning optimization framework. We propose Adjacent Leader Decentralized Gradient Descent (AL-DSGD), for improving final model performance, accelerating convergence, and reduc…
Deep Learning