Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMs
Accurate uncertainty quantification of large language models (LLMs) provides credibility measure over their outputs. However, fine-tuned LLMs often struggle with overconfidence in uncertain predictions due to the limitations in the models' ability to generalize with limited data. Existing parameter efficient fine-tuning (PEFT) uncertainty quantification methods for LLMs focus on post fine-tuning stage and fall short of calibrating epistemic uncertainty. To address these limitations, we propose Functional-Level Uncertainty Quantification for Calibrated Fine-Tuning (UQ4CT), which captures and calibrates epistemic uncertainty over the space of functions that map input prompts to outputs. We implement UQ4CT during the fine-tuning stage via a mixture-of-experts framework that hierarchically decomposes the functional space. We demonstrate that UQ4CT reduces Expected Calibration Error (ECE) by more than $25\%$ while maintaining high accuracy across $5$ benchmarks. Even under distribution shift, UQ4CT maintains superior ECE performance with high accuracy, showcasing improved generalizability.
Code (0)
등록된 구현이 없습니다.
Tasks
Common Sense ReasoningMixture-of-Expertsparameter-efficient fine-tuningUncertainty QuantificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Calibrated Uncertainty Quantification for Operator Learning via Conformal Prediction
Operator learning has been increasingly adopted in scientific and engineering applications, many of which require calibrated uncertainty quantification. Since the output of operator learning is a continuous function, qua…
Conformal PredictionOperator learningPredictionUncertainty QuantificationA Physics inspired Functional Operator for Model Uncertainty Quantification in the RKHS
Accurate uncertainty quantification of model predictions is a crucial problem in machine learning. Existing Bayesian methods, being highly iterative, are expensive to implement and often fail to accurately capture a mode…
Uncertainty QuantificationFast Calibrated Explanations: Efficient and Uncertainty-Aware Explanations for Machine Learning Models
This paper introduces Fast Calibrated Explanations, a method designed for generating rapid, uncertainty-aware explanations for machine learning models. By incorporating perturbation techniques from ConformaSight - a glob…
Computational EfficiencyFeature ImportanceregressionUncertainty QuantificationConcentration and Calibration in Predictive Bayesian Inference
Predictive Bayesian inference (PBI) represents a model-and prior-agnostic approach to standard Bayesian inference which allows users to quantify uncertainty for a functional of interest only by specifying a forward predi…
Bayesian InferenceWhen in Doubt: Neural Non-Parametric Uncertainty Quantification for Epidemic Forecasting
Accurate and trustworthy epidemic forecasting is an important problem that has impact on public health planning and disease mitigation. Most existing epidemic forecasting models disregard uncertainty quantification, resu…
Time SeriesTime Series AnalysisTime Series ForecastingUncertainty Quantification