Generalised Probabilistic Modelling and Improved Uncertainty Estimation in Comparative LLM-as-a-judge
This paper explores generalised probabilistic modelling and uncertainty estimation in comparative LLM-as-a-judge frameworks. We show that existing Product-of-Experts methods are specific cases of a broader framework, enabling diverse modelling options. Furthermore, we propose improved uncertainty estimates for individual comparisons, enabling more efficient selection and achieving strong performance with fewer evaluations. We also introduce a method for estimating overall ranking uncertainty. Finally, we demonstrate that combining absolute and comparative scoring improves performance. Experiments show that the specific expert model has a limited impact on final rankings but our proposed uncertainty estimates, especially the probability of reordering, significantly improve the efficiency of systems reducing the number of needed comparisons by ~50%. Furthermore, ranking-level uncertainty metrics can be used to identify low-performing predictions, where the nature of the probabilistic model has a notable impact on the quality of the overall uncertainty.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
The Multivariate Generalised von Mises distribution: Inference and applications
Circular variables arise in a multitude of data-modelling contexts ranging from robotics to the social sciences, but they have been largely overlooked by the machine learning community. This paper partially redresses thi…
Gaussian ProcessesGeneralised Linear Models for Dependent Binary Outcomes with Applications to Household Stratified Pandemic Influenza Data
Much traditional statistical modelling assumes that the outcome variables of interest are independent of each other when conditioned on the explanatory variables. This assumption is strongly violated in the case of infec…
Model SelectionUncertainty QuantificationEfficient Estimation of the number of neighbours in Probabilistic K Nearest Neighbour Classification
Probabilistic k-nearest neighbour (PKNN) classification has been introduced to improve the performance of original k-nearest neighbour (KNN) classification algorithm by explicitly modelling uncertainty in the classificat…
ClassificationDecision MakingGeneral ClassificationModel SelectionUncertainty Estimation for End-To-End Learned Dense Stereo Matching via Probabilistic Deep Learning
Motivated by the need to identify erroneous disparity assignments, various approaches for uncertainty and confidence estimation of dense stereo matching have been presented in recent years. As in many other fields, espec…
Probabilistic Deep LearningStereo MatchingProbabilistic task modelling for meta-learning
We propose probabilistic task modelling -- a generative probabilistic model for collections of tasks used in meta-learning. The proposed model combines variational auto-encoding and latent Dirichlet allocation to model e…
Meta-Learningparameter estimationVariational Inference