Tipping the Balance: Impact of Class Imbalance Correction on the Performance of Clinical Risk Prediction Models
Objective: ML-based clinical risk prediction models are increasingly used to support decision-making in healthcare. While class-imbalance correction techniques are commonly applied to improve model performance in settings with rare outcomes, their impact on probabilistic calibration remains insufficiently understood. This study evaluated the effect of widely used resampling strategies on both discrimination and calibration across real-world clinical prediction tasks. Methods: Ten clinical datasets spanning diverse medical domains and including 605,842 patients were analyzed. Multiple machine-learning model families, including linear models and several non-linear approaches, were evaluated. Models were trained on the original data and under three commonly used 1:1 class-imbalance correction strategies (SMOTE, RUS, ROS). Performance was assessed on held-out data using discrimination and calibration metrics. Results: Across all datasets and model families, resampling had no positive impact on predictive performance. Changes in the Receiver Operating Characteristic Area Under Curve (ROC-AUC) relative to models trained on the original data were small and inconsistent (ROS: -0.002, p<0.05; RUS: -0.004, p>0.05; SMOTE: -0.01, p<0.05), with no resampling strategy demonstrating a systematic improvement. In contrast, resampling in general degraded the calibration performance. Models trained using imbalance correction exhibited higher Brier scores (0.029 to 0.080, p<0.05), reflecting poorer probabilistic accuracy, and marked deviations in calibration intercept and slope, indicating systematic distortions of predicted risk despite preserved rank-based performance. Conclusion: In a diverse set of real-world clinical prediction tasks, commonly used class-imbalance correction techniques did not provide generalizable improvements in discrimination and were associated with degraded calibration.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Addressing Fairness, Bias and Class Imbalance in Machine Learning: the FBI-loss
Resilience to class imbalance and confounding biases, together with the assurance of fairness guarantees are highly desirable properties of autonomous decision-making systems with real-life impact. Many different targete…
BIG-bench Machine LearningDecision MakingFairnessLet the Fuzzy Rule Speak: Enhancing In-context Learning Debiasing with Interpretability
One of the potential failures of large language models (LLMs) is their imbalanced class performances in text classification tasks. With in-context learning (ICL), LLMs yields good accuracy for some classes but low accura…
In-Context LearningMulti Class Text Classificationtext-classificationText ClassificationTwice Class Bias Correction for Imbalanced Semi-Supervised Learning
Differing from traditional semi-supervised learning, class-imbalanced semi-supervised learning presents two distinct challenges: (1) The imbalanced distribution of training samples leads to model bias towards certain cla…
Bayes Imbalance Impact Index: A Measure of Class Imbalanced Dataset for Classification Problem
Recent studies have shown that imbalance ratio is not the only cause of the performance loss of a classifier in imbalanced data classification. In fact, other data factors, such as small disjuncts, noises and overlapping…
General ClassificationPhased Progressive Learning with Coupling-Regulation-Imbalance Loss for Imbalanced Data Classification
Deep convolutional neural networks often perform poorly when faced with datasets that suffer from quantity imbalances and classification difficulties. Despite advances in the field, existing two-stage approaches still ex…
Classificationimbalanced classificationRepresentation Learning