Noise-Response Calibration: A Causal Intervention Protocol for LLM-Judges
Large language models (LLMs) are increasingly used as automated judges and synthetic labelers, especially in low-label settings. Yet these systems are stochastic and often overconfident, which makes deployment decisions difficult when external ground truth is limited. We propose a practical calibration protocol based on controlled input interventions: if noise severity increases, task performance should exhibit a statistically significant deterioration trend. We operationalize this with a slope-based hypothesis test over repeated trials, using signal-to-noise-ratio (SNR) perturbations for tabular data and lexical perturbations for text data. Across UCI tabular benchmarks and four text classification datasets, we find clear modality-dependent behavior. Our results reveal a modality gap: while text-based judges degrade predictably, the majority of tabular datasets show a lack of statistically significant performance deterioration even under significant signal-to-noise reduction. Interestingly we find that model performance is lower on datasets that are insensitive to noise interventions. We present a reproducible methodology and reporting protocol for robust LLM-judge calibration under distribution shift.
Code (0)
등록된 구현이 없습니다.
Tasks
Text ClassificationSimilar Papers 제목 키워드 기반
Causal Reinforcement Learning for Complex Card Games: A Magic The Gathering Benchmark
Causal reinforcement learning (RL) lacks benchmarks for complex systems that combine sequential decision making, hidden information, large masked action spaces, and explicit causal structure. We introduce MTG-Causal-RL, …
Reinforcement LearningDecision MakingPartial Causal Structure Learning for Valid Selective Conformal Inference under Interventions
Selective conformal prediction can yield substantially tighter uncertainty sets when we can identify calibration examples that are exchangeable with the test example. In interventional settings, such as perturbation expe…
TabCausal: Pretraining Across Causal Environments for Tabular Causal Discovery
Causal discovery aims to recover directed causal relations from observational and interventional data, providing a basis for mechanistic understanding and reliable decision-making. Causal discovery foundation models (CDF…
Amortized Interventional Forecasting for Multivariate CIR Processes
Mean-reverting dynamics are pervasive in finance, and the Cox--Ingersoll--Ross (CIR) process is a standard model for the time series they produce, from short rates to credit default swap (CDS) spreads. Yet CIR models cap…
Cost-effectively Identifying Causal Effect When Only Response Variable Observable
In many real tasks, we care about how to make decisions other than mere predictions on an event, e.g. how to increase the revenue next month instead of knowing it will drop. The key is to identify the causal effects on t…
Decision Making