PFGE: Parsimonious Fast Geometric Ensembling of DNNs
Ensemble methods are commonly used to enhance the generalization performance of machine learning models. However, they present a challenge in deep learning systems due to the high computational overhead required to train an ensemble of deep neural networks (DNNs). Recent advancements such as fast geometric ensembling (FGE) and snapshot ensembles have addressed this issue by training model ensembles in the same time as a single model. Nonetheless, these techniques still require additional memory for test-time inference compared to single-model-based methods. In this paper, we propose a new method called parsimonious FGE (PFGE), which employs a lightweight ensemble of higher-performing DNNs generated through successive stochastic weight averaging procedures. Our experimental results on CIFAR-{10,100} and ImageNet datasets across various modern DNN architectures demonstrate that PFGE achieves 5x memory efficiency compared to previous methods, without compromising on generalization performance. For those interested, our code is available at https://github.com/ZJLAB-AMMI/PFGE.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
The loss functions of deep neural networks are complex and their geometric properties are not well understood. We show that the optima of these complex loss functions are in fact connected by simple curves over which tra…
TabPFGen -- Tabular Data Generation with TabPFN
Advances in deep generative modelling have not translated well to tabular data. We argue that this is caused by a mismatch in structure between popular generative models and discriminative models of tabular data. We thus…
Data AugmentationImputationIn-Context LearningTabular Data GenerationImproved Convex Decomposition with Ensembling and Boolean Primitives
Describing a scene in terms of primitives -- geometrically simple shapes that offer a parsimonious but accurate abstraction of structure -- is an established vision problem. This is a good model of a difficult fitting pr…
regressionScene SegmentationAssessing the Benefits of Model Ensembles in Neural Re-Ranking for Passage Retrieval
Our work aimed at experimentally assessing the benefits of model ensembling within the context of neural methods for passage reranking. Starting from relatively standard neural models, we use a previous technique named F…
Learning-To-RankPassage RerankingPassage RetrievalReranking+2Fast Hyperparameter Optimization of Deep Neural Networks via Ensembling Multiple Surrogates
The performance of deep neural networks crucially depends on good hyperparameter configurations. Bayesian optimization is a powerful framework for optimizing the hyperparameters of DNNs. These methods need sufficient eva…
Bayesian OptimizationHyperparameter Optimization