SCOPE: Sequential Conformal Probing for Reliable OOD Rejection in LLM Services
Rejecting inputs outside the defined in-distribution (IND) service scope is critical for large language model (LLM) services, where unsupported requests should be filtered before full generation. Existing out-of-distribution (OOD) detectors often rely on final outputs or final-layer representations, leaving unclear where service-boundary signals are most clearly encoded inside the model; they also lack a theoretical guarantee for held-out inputs. In this paper, we introduce SCOPE (Sequential Conformal OOD Probing and Evaluation), a framework that selects a readable hidden layer, constructs a conformal gate with IND calibration, and uses a supermartingale e-process to certify persistent service-boundary evidence. Experiments across multiple LLM backbones and six carefully designed boundary conditions show that SCOPE improves gate-level rejection over standard final-layer detectors, while revealing how different OOD boundaries take different geometric forms in hidden space.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Conformal Generative Modeling with Improved Sample Efficiency through Sequential Greedy Filtering
Generative models lack rigorous statistical guarantees for their outputs and are therefore unreliable in safety-critical applications. In this work, we propose Sequential Conformal Prediction for Generative Models (SCOPE…
Conformal PredictionPredictionText GenerationvalidStochastic Submodular Probing with State-Dependent Costs
In this paper, we study a new stochastic submodular maximization problem with state-dependent costs and rejections. The input of our problem is a budget constraint $B$, and a set of items whose states (i.e., the marginal…
SCOPE: Selective Conformal Optimized Pairwise LLM Judging
Large language models (LLMs) are increasingly used as scalable judges in pairwise evaluation, but they remain prone to miscalibration and biases. We propose \textsc{Scope} (Selective Conformal Optimized Pairwise Evaluati…
Everywhere Valid Bounds on False Discovery Proportions in Conformal Inference
Modern applications of conformal inference to multiple testing problems, such as outlier detection and candidate selection, often involve selecting test samples whose conformal p-values fall below a threshold. The qualit…
Outlier DetectionConformal Uncertainty Indicator for Continual Test-Time Adaptation
Continual Test-Time Adaptation (CTTA) aims to adapt models to sequentially changing domains during testing, relying on pseudo-labels for self-adaptation. However, incorrect pseudo-labels can accumulate, leading to perfor…
Conformal PredictionTest-time Adaptation