A Deployment Audit of Release-Side Risk in Conformal Triage under Prevalence Shift
Conformal triage converts predictive scores into deployment actions that either release a case, flag it for urgent attention, or defer it to human review. Under an observed change in target-event prevalence, however, marginal coverage and human-review rate can miss whether patients who experience the target event are released without review. To address this gap, we introduce a leakage-aware deployment audit for release-side conformal triage. It first assigns target subjects to three non-overlapping roles: prevalence correction, conformal calibration, and held-out release-side evaluation. This separation then lets the audit evaluate release directly: how many event-positive patients are cleared without review, whether the pilot has enough event labels for calibration, and how the release-review trade-off shifts. Applying this audit to a retrospective non-small-cell lung cancer (NSCLC) target cohort shows why lower review can be misleading: after prevalence correction, the pooled conformal branch lowers review by releasing more patients, some of whom are event-positive. Within the audit, the classwise branch acts as a scarcity diagnostic: the pilot has too few event labels to support a low-review release rule.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Conformal Selective Acting: Anytime-Valid Risk Control for RLVR-Trained LLMs
A local specialist LLM, fine-tuned with reinforcement learning from verifiable rewards (RLVR) on operator-local data, is installed in a regulated organization with per-deployment error budget $α$. The operator needs a sa…
Reinforcement LearningHybrid Adaptive Conformal Offline Reinforcement Learning for Fair Population Health Management
Population health management programs for Medicaid populations coordinate longitudinal outreach and services (e.g., benefits navigation, behavioral health, social needs support, and clinical scheduling) and must be safe,…
Reinforcement LearningOffline RLStop Shipping AI Agents on Faith: Capability Is Not Production Readiness
AI agents are moving into production workflows where they retrieve information, call tools, maintain state, and act on behalf of users or organizations, but many release decisions still rely on capability signals, demos,…
MedSkillAudit: A Domain-Specific Audit Framework for Medical Research Agent Skills
Background: Agent skills are increasingly deployed as modular, reusable capability units in AI agent systems. Medical research agent skills require safeguards beyond general-purpose evaluation, including scientific integ…
Conformal Calibration: Ensuring the Reliability of Black-Box AI in Wireless Systems
AI is poised to revolutionize telecommunication networks by boosting efficiency, automation, and decision-making. However, the black-box nature of most AI models introduces substantial risk, possibly deterring adoption b…
counterfactualDecision MakingDiagnosticUncertainty Quantification