Balanced Accuracy: The Right Metric for Evaluating LLM Judges -- Explained through Youden's J statistic
Rigorous evaluation of large language models (LLMs) relies on comparing models by the prevalence of desirable or undesirable behaviors, such as task pass rates or policy violations. These prevalence estimates are produced by a classifier, either an LLM-as-a-judge or human annotators, making the choice of classifier central to trustworthy evaluation. Common metrics used for this choice, such as Accuracy, Precision, and F1, are sensitive to class imbalance and to arbitrary choices of positive class, and can favor judges that distort prevalence estimates. We show that Youden's $J$ statistic is theoretically aligned with choosing the best judge to compare models, and that Balanced Accuracy is an equivalent linear transformation of $J$. Through both analytical arguments and empirical examples and simulations, we demonstrate how selecting judges using Balanced Accuracy leads to better, more robust classifier selection.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
LLMs on Trial: Evaluating Judicial Fairness for Large Language Models
Large Language Models (LLMs) are increasingly used in high-stakes fields where their decisions impact rights and equity. However, LLMs' judicial fairness and implications for social justice remain underexplored. When LLM…
Can LLM Assist in the Evaluation of the Quality of Machine Learning Explanations?
EXplainable machine learning (XML) has recently emerged to address the mystery mechanisms of machine learning (ML) systems by interpreting their 'black box' results. Despite the development of various explanation methods…
Language ModelingLanguage ModellingLarge Language ModelTuning LLM Judges Hyperparameters
Evaluating Large Language Models (LLMs) often requires costly human annotations. To address this, LLM-based judges have been proposed, which compare the outputs of two LLMs enabling the ranking of models without human in…
Was Javert right to be suspicious? Marginal Treatment Effects with Duration Outcomes
We identify the distributional and quantile marginal treatment effect functions when the outcome is right-censored. Our method requires a conditionally exogenous instrument and random censoring. We propose asymptotically…
validMetric Match: A Subset Selection Approach to Evaluating LLM Judge Reliability
LLM judges are used to reduce the need for costly human labor in evaluating open-ended text generation. However, the reliability of these judges depends critically on their alignment with human raters -- a property that …
Text Generation