Distributed Continual Learning
This work studies the intersection of continual and federated learning, in which independent agents face unique tasks in their environments and incrementally develop and share knowledge. We introduce a mathematical framework capturing the essential aspects of distributed continual learning, including agent model and statistical heterogeneity, continual distribution shift, network topology, and communication constraints. Operating on the thesis that distributed continual learning enhances individual agent performance over single-agent learning, we identify three modes of information exchange: data instances, full model parameters, and modular (partial) model parameters. We develop algorithms for each sharing mode and conduct extensive empirical investigations across various datasets, topology structures, and communication limits. Our findings reveal three key insights: sharing parameters is more efficient than sharing data as tasks become more complex; modular parameter sharing yields the best performance while minimizing communication costs; and combining sharing modes can cumulatively improve performance.
Code (0)
등록된 구현이 없습니다.
Tasks
Continual LearningFederated LearningSimilar Papers 제목 키워드 기반
Sparse Distributed Memory is a Continual Learner
Continual learning is a problem for artificial neural networks that their biological counterparts are adept at solving. Building on work using Sparse Distributed Memory (SDM) to connect a core neural circuit with the pow…
Continual LearningContinual Learning with Distributed Optimization: Does CoCoA Forget?
We focus on the continual learning problem where the tasks arrive sequentially and the aim is to perform well on the newly arrived task without performance degradation on the previously seen tasks. In contrast to the con…
Continual LearningDistributed OptimizationContinual Distributed Learning for Crisis Management
Social media platforms such as Twitter, Facebook etc can be utilised as an important source of information during disaster events. This information can be used for disaster response and crisis management if processed acc…
Continual LearningDisaster ResponseFederated LearningManagement+2Distributed Continual Learning with CoCoA in High-dimensional Linear Regression
We consider estimation under scenarios where the signals of interest exhibit change of characteristics over time. In particular, we consider the continual learning problem where different tasks, e.g., data with different…
Continual LearningregressionUnleashing the Power of Continual Learning on Non-Centralized Devices: A Survey
Non-Centralized Continual Learning (NCCL) has become an emerging paradigm for enabling distributed devices such as vehicles and servers to handle streaming data from a joint non-stationary environment. To achieve high re…
Continual Learning