Estimating the Conformal Prediction Threshold from Noisy Labels

2025-01-22 · Coby Penso, Jacob Goldberger, Ethan Fetaya

Conformal Prediction (CP) is a method to control prediction uncertainty by producing a small prediction set, ensuring a predetermined probability that the true class lies within this set. This is commonly done by defining a score, based on the model predictions, and setting a threshold on this score using a validation set. In this study, we address the problem of CP calibration when we only have access to a validation set with noisy labels. We show how we can estimate the noise-free conformal threshold based on the noisy labeled data. Our solution is flexible and can accommodate various modeling assumptions regarding the label contamination process, without needing any information about the underlying data distribution or the internal mechanisms of the machine learning classifier. We develop a coverage guarantee for uniform noise that is effective even in tasks with a large number of classes. We dub our approach Noise-Aware Conformal Prediction (NACP) and show on several natural and medical image classification datasets, including ImageNet, that it significantly outperforms current noisy label methods and achieves results comparable to those obtained with a clean validation set.

📄 PDF Abstract BibTeX arXiv:2501.12749

Code (1)

cobypenso/noise-aware-conformal-prediction 공식 구현 pytorch

Tasks

Conformal Predictionimage-classification Image Classification Medical Image Classification Prediction

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Efficient Conformal Prediction for Regression Models under Label Noise

2025-09-18 · Yahav Cohen, Jacob Goldberger, Tom Tirer arxiv

In high-stakes scenarios, such as medical imaging applications, it is critical to equip the predictions of a regression model with reliable confidence intervals. Recently, Conformal Prediction (CP) has emerged as a power…

Label Noise Robustness of Conformal Prediction

2022-09-28 · Bat-Sheva Einbinder, Shai Feldman, Stephen Bates, Anastasios N. Angelopoulos 외

We study the robustness of conformal prediction, a powerful tool for uncertainty quantification, to label noise. Our analysis tackles both regression and classification problems, characterizing when and how it is possibl…

Conformal Prediction Prediction regression Uncertainty Quantification

Privacy-Preserving Conformal Prediction Under Local Differential Privacy

2025-05-21 · Coby Penso, Bar Mahpud, Jacob Goldberger, Or Sheffet

Conformal prediction (CP) provides sets of candidate classes with a guaranteed probability of containing the true class. However, it typically relies on a calibration set with clean labels. We address privacy-sensitive s…

Conformal Prediction Large Language Model Prediction Privacy Preserving

Conformal Thresholded Intervals for Efficient Regression

2024-07-19 · Rui Luo, Zhixin Zhou

This paper introduces Conformal Thresholded Intervals (CTI), a novel conformal regression method that aims to produce the smallest possible prediction set with guaranteed coverage. Unlike existing methods that rely on ne…

Conformal Prediction Prediction quantile regression regression+1

Conformal Margin Risk Minimization: An Envelope Framework for Robust Learning under Label Noise

2026-04-07 · Yuanjie Shi, Peihong Li, Zijian Zhang, Janardhan Rao Doppa 외 arxiv

Most methods for learning with noisy labels require privileged knowledge such as noise transition matrices, clean subsets or pretrained feature extractors, resources typically unavailable when robustness is most needed. …

Learning with noisy labels