Data-driven Error Estimation: Upper Bounding Multiple Errors with No Technical Debt
We formulate the problem of constructing multiple simultaneously valid confidence intervals (CIs) as estimating a high probability upper bound on the maximum error for a class/set of estimate-estimand-error tuples, and refer to this as the error estimation problem. For a single such tuple, data-driven confidence intervals can often be used to bound the error in our estimate. However, for a class of estimate-estimand-error tuples, nontrivial high probability upper bounds on the maximum error often require class complexity as input -- limiting the practicality of such methods and often resulting in loose bounds. Rather than deriving theoretical class complexity-based bounds, we propose a completely data-driven approach to estimate an upper bound on the maximum error. The simple and general nature of our solution to this fundamental challenge lends itself to several applications including: multiple CI construction, multiple hypothesis testing, estimating excess risk bounds (a fundamental measure of uncertainty in machine learning) for any training/fine-tuning algorithm, and enabling the development of a contextual bandit pipeline that can leverage any reward model estimation procedure as input (without additional mathematical analysis).
Code (0)
등록된 구현이 없습니다.
Tasks
validSimilar Papers 제목 키워드 기반
Rate-matching the regret lower-bound in the linear quadratic regulator with unknown dynamics
The theory of reinforcement learning currently suffers from a mismatch between its empirical performance and the theoretical characterization of its performance, with consequences for, e.g., the understanding of sample e…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Tight Bounds for Uncertain Time-Correlated Errors with Gauss-Markov Structure
Safety-critical navigation applications require that estimation errors be reliably quantified and bounded. This can be challenging for linear dynamic systems if the process noise or measurement errors have uncertain time…
Rademacher upper bounds for cross-validation errors with an application to the lasso
We establish a general upper bound for $K$-fold cross-validation ($K$-CV) errors that can be adapted to many $K$-CV-based estimators and learning algorithms. Based on Rademacher complexity of the model and the Orlicz-$\P…
BlockingVariable SelectionStructured Estimation with Atomic Norms: General Bounds and Applications
For structured estimation problems with atomic norms, recent advances in the literature express sample complexity and estimation error bounds in terms of certain geometric measures, in particular Gaussian width of the un…
A Note on Bounding Regret of the C$^2$UCB Contextual Combinatorial Bandit
We revisit the proof by Qin et al. (2014) of bounded regret of the C$^2$UCB contextual combinatorial bandit. We demonstrate an error in the proof of volumetric expansion of the moment matrix, used in upper bounding a fun…