Correcting Performance Estimation Bias in Imbalanced Classification with Minority Subconcepts
Class-level evaluation can conceal substantial performance disparities across subconcepts within the same class, causing models that perform well on average to fail on specific subpopulations. Prior work has shown that common evaluation measures for imbalanced classification are biased toward larger minority subconcepts and that utility-based reweighting using true subconcept labels can mitigate this bias; however, such labels are rarely available at test time. We introduce a practical utility-weighted evaluation that replaces unavailable subconcept labels with predicted posterior probabilities from a multiclass subconcept model. Evaluation weights are defined as the expected utility under this posterior, yielding a soft, uncertainty-aware metric we call predicted-weighted balanced accuracy (pBA). Experiments on tabular benchmarks as well as medical-imaging and text datasets show that unweighted scores can be misleading under within-class heterogeneity, while pBA provides more stable and interpretable assessments when subconcept distributions are uneven but not pathological. Our code is available at: https://anonymous.4open.science/r/correcting-bias-imbalance-9C6C/.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
DP-Adam: Correcting DP Bias in Adam's Second Moment Estimation
We observe that the traditional use of DP with the Adam optimizer introduces a bias in the second moment estimation, due to the addition of independent noise in the gradient computation. This bias leads to a different sc…
Bayesian Imbalanced Regression Debiasing
Imbalanced regression, where the training data has an uneven distribution on its range, is widely encountered in the real world, e.g., age estimation (uni-dimensional regression) and pose estimation (multi-dimensional re…
Age Estimationimbalanced classificationPose EstimationregressionYour Group-Relative Advantage Is Biased
Reinforcement Learning from Verifier Rewards (RLVR) has emerged as a widely used approach for post-training large language models on reasoning tasks, with group-based methods such as GRPO and its variants gaining broad a…
Reinforcement LearningMathematical ReasoningBias-Corrected Data Synthesis for Imbalanced Learning
Imbalanced data, where the positive samples represent only a small proportion compared to the negative samples, makes it challenging for classification problems to balance the false positive and false negative rates. A c…
Multi-Task LearningCausal InferenceSelf-correcting Q-Learning
The Q-learning algorithm is known to be affected by the maximization bias, i.e. the systematic overestimation of action values, an important issue that has recently received renewed attention. Double Q-learning has been …
Q-Learning