Uncertainty Quantification of Large Language Models using Approximate Bayesian Computation
Despite their widespread applications, Large Language Models (LLMs) often struggle to express uncertainty, posing a challenge for reliable deployment in high stakes and safety critical domains like clinical diagnostics. Existing standard baseline methods such as model logits and elicited probabilities produce overconfident and poorly calibrated estimates. In this work, we propose Approximate Bayesian Computation (ABC), a likelihood-free Bayesian inference, based approach that treats LLMs as a stochastic simulator to infer posterior distributions over predictive probabilities. We evaluate our ABC approach on two clinically relevant benchmarks: a synthetic oral lesion diagnosis dataset and the publicly available GretelAI symptom-to-diagnosis dataset. Compared to standard baselines, our approach improves accuracy by up to 46.9\%, reduces Brier scores by 74.4\%, and enhances calibration as measured by Expected Calibration Error (ECE) and predictive entropy.
Code (0)
등록된 구현이 없습니다.
Tasks
Bayesian InferenceSimilar Papers 제목 키워드 기반
Approximate Bayesian Neural Operators: Uncertainty Quantification for Parametric PDEs
Neural operators are a type of deep architecture that learns to solve (i.e. learns the nonlinear solution operator of) partial differential equations (PDEs). The current state of the art for these models does not provide…
Gaussian ProcessesUncertainty QuantificationBayesian BiLO: Bilevel Local Operator Learning for Efficient Uncertainty Quantification of Bayesian PDE Inverse Problems with Low-Rank Adaptation
Uncertainty quantification in PDE inverse problems is essential in many applications. Scientific machine learning and AI enable data-driven learning of model components while preserving physical structure, and provide th…
Bayesian InferenceDeep Ensemble as a Gaussian Process Approximate Posterior
Deep Ensemble (DE) is an effective alternative to Bayesian neural networks for uncertainty quantification in deep learning. The uncertainty of DE is usually conveyed by the functional inconsistency among the ensemble mem…
Bayesian InferenceUncertainty QuantificationDeep Ensembles from a Bayesian Perspective
Deep ensembles can be considered as the current state-of-the-art for uncertainty quantification in deep learning. While the approach was originally proposed as a non-Bayesian technique, arguments supporting its Bayesian …
Uncertainty QuantificationBatch simulations and uncertainty quantification in Gaussian process surrogate-based approximate Bayesian computation
Surrogate models can be used to accelerate approximate Bayesian computation (ABC). In one such framework the discrepancy between simulated and observed data is modelled with a Gaussian process. So far principled strategi…
Uncertainty Quantification