Score-based calibration testing for multivariate forecast distributions
Calibration tests based on the probability integral transform (PIT) are routinely used to assess the quality of univariate distributional forecasts. However, PIT-based calibration tests for multivariate distributional forecasts face various challenges. We propose two new types of tests based on proper scoring rules, which overcome these challenges. They arise from a general framework for calibration testing in the multivariate case, introduced in this work. The new tests have good size and power properties in simulations and solve various problems of existing tests. We apply the tests to forecast distributions for macroeconomic and financial time series data.
Code (1)
Tasks
Time SeriesTime Series AnalysisSimilar Papers 제목 키워드 기반
MVG-CRPS: A Robust Loss Function for Multivariate Probabilistic Forecasting
Multivariate Gaussian (MVG) distributions are central to modeling correlated continuous variables in probabilistic forecasting. Neural forecasting models typically parameterize the mean vector and covariance matrix of th…
Probabilistic Time Series Forecastingscoring ruleTime Series ForecastingUncertainty QuantificationCalibrated Multivariate Distributional Regression with Pre-Rank Regularization
The goal of probabilistic prediction is to issue predictive distributions that are as informative as possible, subject to being calibrated. Despite substantial progress in the univariate setting, achieving multivariate c…
Recipes for Calibration Checks in Safety-Critical Applications
Safety-critical prediction systems, such as autonomous vehicles, weather forecasters, and medical monitors, commonly rely on probabilistic forecasters. These forecasters make predictions about possible future outcomes, a…
Robot Pose EstimationAutonomous VehiclesWeather ForecastingDecision MakingCalibrated Multivariate Regression with Localized PIT Mappings
Calibration ensures that predicted uncertainties align with observed uncertainties. While there is an extensive literature on recalibration methods for univariate probabilistic forecasts, work on calibration for multivar…
regressionBeyond Uncertainty Sets: Leveraging Optimal Transport to Extend Conformal Predictive Distribution to Multivariate Settings
Conformal prediction (CP) constructs uncertainty sets for model outputs with finite-sample coverage guarantees. A candidate output is included in the prediction set if its non-conformity score is not considered extreme r…