FastEnsemble: Benchmarking and Accelerating Ensemble-based Uncertainty Estimation for Image-to-Image Translation
Estimating prediction uncertainty and confidence of deep learning models is crucial for mission-critical machine learning applications, such as biomedical imaging for diagnostics or therapy, and self-driving cars. However, making robust uncertainty estimation is complicated given the variety of learning objectives, data modalities, types of data corruption. Previous studies often addressed such a challenge by restricting datasets to standard ones like CIFAR or ImageNet. While convenient, it is doubtful whether the same conclusion holds for real-life datasets, in which more complicated image generation tasks are involved. This paper presents a different perspective to evaluate how confidence and uncertainty estimators behave under distribution shifts, focusing on the biomedical imaging domain. Specifically, we test a series of pair-wise cell imaging datasets using a new metric to compare existing models. In addition, we introduce FastEnsemble, a fast ensemble method which only requires less than $8\%$ of the full-ensemble training time to generate a new ensemble member. Our experiments show that the proposed fast ensemble method is able to substantially improve the speed vs. quality trade-off.
Code (0)
등록된 구현이 없습니다.
Tasks
BenchmarkingImage GenerationImage-to-Image TranslationSelf-Driving CarsTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control
Uncertainty quantification is one of the central challenges for machine learning in real-world applications. In reinforcement learning, an agent confronts two kinds of uncertainty, called epistemic uncertainty and aleato…
continuous-controlContinuous Controlreinforcement-learningReinforcement Learning+2Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning
Offline reinforcement learning agents seek optimal policies from fixed data sets. With environmental interaction prohibited, agents face significant challenges in preventing errors in value estimates from compounding and…
Behavioural cloningBenchmarkingreinforcement-learningEvaluating Prediction Uncertainty Estimates from BatchEnsemble
Deep learning models struggle with uncertainty estimation. Many approaches are either computationally infeasible or underestimate uncertainty. We investigate \textit{BatchEnsemble} as a general and scalable method for un…
Benchmarking common uncertainty estimation methods with histopathological images under domain shift and label noise
In the past years, deep learning has seen an increase in usage in the domain of histopathological applications. However, while these approaches have shown great potential, in high-risk environments deep learning models n…
BenchmarkingClassificationData AugmentationVariational Inference+1Packed-Ensemble Surrogate Models for Fluid Flow Estimation Arround Airfoil Geometries
Physical based simulations can be very time and computationally demanding tasks. One way of accelerating these processes is by making use of data-driven surrogate models that learn from existing simulations. Ensembling m…