How Good is a Single Basin?
The multi-modal nature of neural loss landscapes is often considered to be the main driver behind the empirical success of deep ensembles. In this work, we probe this belief by constructing various "connected" ensembles which are restricted to lie in the same basin. Through our experiments, we demonstrate that increased connectivity indeed negatively impacts performance. However, when incorporating the knowledge from other basins implicitly through distillation, we show that the gap in performance can be mitigated by re-discovering (multi-basin) deep ensembles within a single basin. Thus, we conjecture that while the extra-basin knowledge is at least partially present in any given basin, it cannot be easily harnessed without learning it from other basins.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Performance Analysis of Basin Hopping Compared to Established Metaheuristics for Global Optimization
During the last decades many metaheuristics for global numerical optimization have been proposed. Among them, Basin Hopping is very simple and straightforward to implement, although rarely used outside its original Physi…
global-optimizationImage Segmentation by Size-Dependent Single Linkage Clustering of a Watershed Basin Graph
We present a method for hierarchical image segmentation that defines a disaffinity graph on the image, over-segments it into watershed basins, defines a new graph on the basins, and then merges basins with a modified, si…
ClusteringImage SegmentationSemantic SegmentationWASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Average
The performance of deep neural networks is enhanced by ensemble methods, which average the output of several models. However, this comes at an increased cost at inference. Weight averaging methods aim at balancing the ge…
image-classificationImage ClassificationTo Stay or Not to Stay in the Pre-train Basin: Insights on Ensembling in Transfer Learning
Transfer learning and ensembling are two popular techniques for improving the performance and robustness of neural networks. Due to the high cost of pre-training, ensembles of models fine-tuned from a single pre-trained …
DiversityTransfer LearningExplicitly Multi-Modal Benchmarks for Multi-Objective Optimization
In multi-objective optimization, designing good benchmark problems is an important issue for improving solvers. Controlling the global location of Pareto optima in existing benchmark problems has been problematic, and it…
Benchmarking