Guaranteed Validity for Empirical Approaches to Adaptive Data Analysis
We design a general framework for answering adaptive statistical queries that focuses on providing explicit confidence intervals along with point estimates. Prior work in this area has either focused on providing tight confidence intervals for specific analyses, or providing general worst-case bounds for point estimates. Unfortunately, as we observe, these worst-case bounds are loose in many settings --- often not even beating simple baselines like sample splitting. Our main contribution is to design a framework for providing valid, instance-specific confidence intervals for point estimates that can be generated by heuristics. When paired with good heuristics, this method gives guarantees that are orders of magnitude better than the best worst-case bounds. We provide a Python library implementing our method.
Code (1)
Tasks
validSimilar Papers 제목 키워드 기반
Adaptive Conformal Prediction for Quantum Machine Learning
Quantum machine learning seeks to leverage quantum computers to improve upon classical machine learning algorithms. Currently, robust uncertainty quantification methods remain underdeveloped in the quantum domain, despit…
Quantum Machine LearningCoverage-Guaranteed Prediction Sets for Out-of-Distribution Data
Out-of-distribution (OOD) generalization has attracted increasing research attention in recent years, due to its promising experimental results in real-world applications. In this paper,we study the confidence set predic…
Conformal PredictionPredictionA Closer Look at the Application of Causal Inference in Graph Representation Learning
Modeling causal relationships in graph representation learning remains a fundamental challenge. Existing approaches often draw on theories and methods from causal inference to identify causal subgraphs or mitigate confou…
Graph Representation LearningCausal InferenceGraph LearningAn Experimental Design for Anytime-Valid Causal Inference on Multi-Armed Bandits
Experimentation is crucial for managers to rigorously quantify the value of a change and determine if it leads to a statistically significant improvement over the status quo. As companies increasingly mandate that all ch…
Causal InferenceExperimental DesignMulti-Armed BanditsvalidHard Constraints Meet Soft Generation: Guaranteed Feasibility for LLM-based Combinatorial Optimization
Large language models (LLMs) have emerged as promising general-purpose solvers for combinatorial optimization (CO), yet they fundamentally lack mechanisms to guarantee solution feasibility which is critical for real-worl…