COM-BOM: Bayesian Exemplar Search for Efficiently Exploring the Accuracy-Calibration Pareto Frontier
Selecting an optimal set of exemplars is critical for good performance of in-context learning. However, prior exemplar search methods narrowly optimize for predictive accuracy, critically neglecting model calibration--a key determinant of trustworthiness and safe deployment. In this paper, we formulate exemplar selection as a multi-objective optimization problem, explicitly targeting both the maximization of predictive accuracy and the minimization of expected calibration error. We solve this problem with a sample-efficient Combinatorial Bayesian Optimization algorithm (COM-BOM) to find the Pareto front that optimally trades off the two objectives of accuracy and calibration. We evaluate COM-BOM on multiple tasks from unsaturated MMLU-Pro benchmark and find that COM-BOM beats or matches the baselines at jointly optimizing the two objectives, while requiring a minimal number of LLM API calls.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
BayesAdapter: Being Bayesian, Inexpensively and Reliably, via Bayesian Fine-tuning
Despite their theoretical appealingness, Bayesian neural networks (BNNs) are left behind in real-world adoption, mainly due to persistent concerns on their scalability, accessibility, and reliability. In this work, we de…
Variational InferenceAdaCL:Adaptive Continual Learning
Class-Incremental Learning aims to update a deep classifier to learn new categories while maintaining or improving its accuracy on previously observed classes. Common methods to prevent forgetting previously learned clas…
Bayesian Optimizationclass-incremental learningClass Incremental LearningContinual Learning+1Constrained Bayesian Optimization under Bivariate Gaussian Process with Application to Cure Process Optimization
Bayesian Optimization, leveraging Gaussian process models, has proven to be a powerful tool for minimizing expensive-to-evaluate objective functions by efficiently exploring the search space. Extensions such as constrain…
Bayesian OptimizationGaussian ProcessesLarge-scale Bayesian Structure Learning for Gaussian Graphical Models using Marginal Pseudo-likelihood
Bayesian methods for learning Gaussian graphical models offer a comprehensive framework that addresses model uncertainty and incorporates prior knowledge. Despite their theoretical strengths, the applicability of Bayesia…
Computational EfficiencyByPE-VAE: Bayesian Pseudocoresets Exemplar VAE
Recent studies show that advanced priors play a major role in deep generative models. Exemplar VAE, as a variant of VAE with an exemplar-based prior, has achieved impressive results. However, due to the nature of model d…
Data AugmentationDensity EstimationRepresentation LearningStochastic Optimization