Towards E-Value Based Stopping Rules for Bayesian Deep Ensembles
Bayesian Deep Ensembles (BDEs) represent a powerful approach for uncertainty quantification in deep learning, combining the robustness of Deep Ensembles (DEs) with flexible multi-chain MCMC. While DEs are affordable in most deep learning settings, (long) sampling of Bayesian neural networks can be prohibitively costly. Yet, adding sampling after optimizing the DEs has been shown to yield significant improvements. This leaves a critical practical question: How long should the sequential sampling process continue to yield significant improvements over the initial optimized DE baseline? To tackle this question, we propose a stopping rule based on E-values. We formulate the ensemble construction as a sequential anytime-valid hypothesis test, providing a principled way to decide whether or not to reject the null hypothesis that MCMC offers no improvement over a strong baseline, to early stop the sampling. Empirically, we study this approach for diverse settings. Our results demonstrate the efficacy of our approach and reveal that only a fraction of the full-chain budget is often required.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Stopping Bayesian Optimization with Probabilistic Regret Bounds
Bayesian optimization is a popular framework for efficiently tackling black-box search problems. As a rule, these algorithms operate by iteratively choosing what to evaluate next until some predefined budget has been exh…
Bayesian OptimizationOn Uniform, Bayesian, and PAC-Bayesian Deep Ensembles
It is common practice to combine deep neural networks into ensembles. These deep ensembles can benefit from the cancellation of errors effect: Errors by ensemble members may average out, leading to better generalization …
Implementability of Honest Multi-Agent Sequential Decision-Making with Dynamic Population
We study the design of decision-making mechanism for resource allocations over a multi-agent system in a dynamic environment. Agents' privately observed preference over resources evolves over time and the population is d…
Decision MakingSequential Decision MakingOptimal Stopping Theory for a Distributionally Robust Seller
Sellers in online markets face the challenge of determining the right time to sell in view of uncertain future offers. Classical stopping theory assumes that sellers have full knowledge of the value distributions, and le…
A stochastic approach to handle knapsack problems in the creation of ensembles
Ensemble-based methods are highly popular approaches that increase the accuracy of a decision by aggregating the opinions of individual voters. The common point is to maximize accuracy; however, a natural limitation occu…