QuantSightBench: Evaluating LLM Quantitative Forecasting with Prediction Intervals
Forecasting has become a natural benchmark for reasoning under uncertainty. Yet existing evaluations of large language models remain limited to judgmental tasks in simple formats, such as binary or multiple-choice questions. In practice, however, forecasting spans a far broader scope. Across domains such as economics, public health, and social demographics, decisions hinge on numerical estimates over continuous quantities, a capability that current benchmarks do not capture. Evaluating such estimates requires a format that makes uncertainty explicit and testable. We propose prediction intervals as a natural and rigorous interface for this purpose. They demand scale awareness, internal consistency across confidence levels, and calibration over a continuum of outcomes, making them a more suitable evaluation format than point estimates for numerical forecasting. To assess this capability, we introduce a new benchmark QuantSightBench, and evaluate frontier models under multiple settings, assessing both empirical coverage and interval sharpness. Our results show that none of the 11 evaluated frontier and open-weight models achieves the 90\% coverage target, with the top performers Gemini 3.1 Pro (79.1\%), Grok 4 (76.4\%), and GPT-5.4 (75.3\%) all falling at least 10 percentage points short. Calibration degrades sharply at extreme magnitudes, revealing systematic overconfidence across all evaluated models.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Prediction intervals for economic fixed-event forecasts
The fixed-event forecasting setup is common in economic policy. It involves a sequence of forecasts of the same (`fixed') predictand, so that the difficulty of the forecasting problem decreases over time. Fixed-event poi…
PredictionPrediction IntervalsregressionDemand Forecasting of Individual Probability Density Functions with Machine Learning
Demand forecasting is a central component of the replenishment process for retailers, as it provides crucial input for subsequent decision making like ordering processes. In contrast to point estimates, such as the condi…
BIG-bench Machine LearningDecision MakingDemand ForecastingTime Series Analysis+1Conformal Prediction for Electricity Price Forecasting in the Day-Ahead and Real-Time Balancing Market
The integration of renewable energy into electricity markets poses significant challenges to price stability and increases the complexity of market operations. Accurate and reliable electricity price forecasting is cruci…
Conformal PredictionDecision Makingenergy tradingPrediction+2Uncertainty-Aware Solar Flare Regression
Current solar flare predictions often lack precise quantification of their reliability, resulting in frequent false alarms, particularly when dealing with datasets skewed towards extreme events. To improve the trustworth…
Weather ForecastingEnhancing reliability in prediction intervals using point forecasters: Heteroscedastic Quantile Regression and Width-Adaptive Conformal Inference
Constructing prediction intervals for time series forecasting is challenging, particularly when practitioners rely solely on point forecasts. While previous research has focused on creating increasingly efficient interva…
PredictionPrediction Intervalsquantile regressionTime Series+2