Model Aggregation via Good-Enough Model Spaces
In many applications, the training data for a machine learning task is partitioned across multiple nodes, and aggregating this data may be infeasible due to communication, privacy, or storage constraints. Existing distributed optimization methods for learning global models in these settings typically aggregate local updates from each node in an iterative fashion. However, these approaches require many rounds of communication between nodes, and assume that updates can be synchronously shared across a connected network. In this work, we present Good-Enough Model Spaces (GEMS), a novel framework for learning a global model by carefully intersecting the sets of "good-enough" models across each node. Our approach utilizes minimal communication and does not require sharing of data between nodes. We present methods for learning both convex models and neural networks within this framework and discuss how small samples of held-out data can be used for post-learning fine-tuning. In experiments on image and medical datasets, our approach on average improves upon other baseline aggregation techniques such as ensembling or model averaging by as much as 15 points (accuracy).
Code (0)
등록된 구현이 없습니다.
Tasks
Distributed OptimizationmodelSentiment AnalysisSimilar Papers 제목 키워드 기반
Model Aggregation via Good-Enough Model Spaces
In many applications, the training data for a machine learning task is partitioned across multiple nodes, and aggregating this data may be infeasible due to storage, communication, or privacy constraints. In this work,…
modelHow good is good enough? Re-evaluating the bar for energy disaggregation
Since the early 1980s, the research community has developed ever more sophisticated algorithms for the problem of energy disaggregation, but despite decades of research, there is still a dearth of applications with demon…
What Makes Language Models Good-enough?
Psycholinguistic research suggests that humans may build a representation of linguistic input that is 'good-enough' for the task at hand. This study examines what architectural features make language models learn human-l…
Renderers are Good Zero-Shot Representation Learners: Exploring Diffusion Latents for Metric Learning
Can the latent spaces of modern generative neural rendering models serve as representations for 3D-aware discriminative visual understanding tasks? We use retrieval as a proxy for measuring the metric learning properties…
Metric LearningNeural RenderingRetrievalA diversity-enhanced genetic algorithm for efficient exploration of parameter spaces
We present a Python package together with a practical guide for the implementation of a lightweight diversity-enhanced genetic algorithm (GA) approach for the exploration of multi-dimensional parameter spaces. Searching …
DiversityEfficient Exploration