Validity problems in clinical machine learning by indirect data labeling using consensus definitions
We demonstrate a validity problem of machine learning in the vital application area of disease diagnosis in medicine. It arises when target labels in training data are determined by an indirect measurement, and the fundamental measurements needed to determine this indirect measurement are included in the input data representation. Machine learning models trained on this data will learn nothing else but to exactly reconstruct the known target definition. Such models show perfect performance on similarly constructed test data but will fail catastrophically on real-world examples where the defining fundamental measurements are not or only incompletely available. We present a general procedure allowing identification of problematic datasets and black-box machine learning models trained on them, and exemplify our detection procedure on the task of early prediction of sepsis.
Code (1)
Similar Papers 제목 키워드 기반
Doubly Robust Estimation of Direct and Indirect Quantile Treatment Effects with Machine Learning
We suggest double/debiased machine learning estimators of direct and indirect quantile treatment effects under a selection-on-observables assumption. This permits disentangling the causal effect of a binary treatment at …
Principles from Clinical Research for NLP Model Generalization
The NLP community typically relies on performance of a model on a held-out test set to assess generalization. Performance drops observed in datasets outside of official test sets are generally attributed to "out-of-distr…
modelRelation ExtractionCausal Machine Learning Is Not a Panacea: A Roadmap for Observational Causal Inference in Health
Objective: The growing availability of large-scale observational clinical datasets and challenges in conducting randomized controlled trials have spurred enthusiasm in using causal machine learning (ML) for causal infere…
Causal InferenceClinically Calibrated Machine Learning Benchmarks for Large-Scale Multi-Disorder EEG Classification
Clinical electroencephalography is routinely used to evaluate patients with diverse and often overlapping neurological conditions, yet interpretation remains manual, time-intensive, and variable across experts. While aut…
Feature ImportanceSeizure DetectionApproximation to Object Conditional Validity with Inductive Conformal Predictors
Conformal predictors are machine learning algorithms that output prediction sets that have a guarantee of marginal validity for finite samples with minimal distributional assumptions. This is a property that makes confor…
BIG-bench Machine LearningObjectPredictionPrediction Intervals+1