Distribution-Free Uncertainty Quantification for Continuous AI Agent Evaluation

2026-05-19 · Yuxuan Gao, Megan Wang, Yi Ling Yu arxiv

We adapt split conformal prediction and adaptive conformal inference (ACI) to continuous AI agent evaluation, providing distribution-free coverage guarantees for forecasted quality scores. Conformal intervals achieve calibration error below 0.02 across all nominal levels at the 24h horizon, while ACI correctly widens intervals by 35% following agent releases then reconverges. We further develop compositional uncertainty bounds for multi-agent pipelines (validated via simulation across inter-stage correlations rho in [-0.5, 0.9]), a conformal abstention rule for pairwise rankings with controlled false-ranking rate, and FDR-corrected abstention for leaderboard-scale multiple testing. Evaluating 50 agents via 18 real-time signals collected hourly, we show that per-agent conditional coverage is well-concentrated around the nominal level (mean 80.4%, 90% of agents within [72%, 90%]), and that cross-source sentiment divergence predicts ranking instability (r=0.64, p<0.01). A circularity-controlled validation confirms the framework captures signal beyond benchmarks (rho_s=0.52, p<0.01, n=35). Code and data are released under CC BY 4.0.

📄 PDF Abstract BibTeX arXiv:2605.19779

Code (0)

등록된 구현이 없습니다.

Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control

2022-07-27 · Takuya Kanazawa, HaiYan Wang, Chetan Gupta

Uncertainty quantification is one of the central challenges for machine learning in real-world applications. In reinforcement learning, an agent confronts two kinds of uncertainty, called epistemic uncertainty and aleato…

continuous-control Continuous Control reinforcement-learning Reinforcement Learning+2

Agentic Uncertainty Quantification

2026-01-22 · Jiaxin Zhang, Prafulla Kumar Choubey, Kung-Hsiang Huang, Caiming Xiong 외 arxiv

Although AI agents have demonstrated impressive capabilities in long-horizon reasoning, their reliability is severely hampered by the ``Spiral of Hallucination,'' where early epistemic errors propagate irreversibly. Exis…

Conformalized-DeepONet: A Distribution-Free Framework for Uncertainty Quantification in Deep Operator Networks

2024-02-23 · Christian Moya, Amirhossein Mollaali, Zecheng Zhang, Lu Lu 외

In this paper, we adopt conformal prediction, a distribution-free uncertainty quantification (UQ) framework, to obtain confidence prediction intervals with coverage guarantees for Deep Operator Network (DeepONet) regress…

Conformal Prediction Prediction Prediction Intervals regression+1

Conformal Contraction for Robust Nonlinear Control with Distribution-Free Uncertainty Quantification

2025-07-18 · Sihang Wei, Melkior Ornik, Hiroyasu Tsukamoto arxiv

We present a novel robust control framework for continuous-time, perturbed nonlinear dynamical systems with uncertainty that depends nonlinearly on both the state and control inputs. Unlike conventional approaches that i…

Motion Planning

Strategic Conformal Prediction

2024-11-03 · Daniel Csillag, Claudio José Struchiner, Guilherme Tegoni Goedert

When a machine learning model is deployed, its predictions can alter its environment, as better informed agents strategize to suit their own interests. With such alterations in mind, existing approaches to uncertainty qu…

Conformal Prediction Prediction Uncertainty Quantification