Oops, I Sampled it Again: Reinterpreting Confidence Intervals in Few-Shot Learning
The predominant method for computing confidence intervals (CI) in few-shot learning (FSL) is based on sampling the tasks with replacement, i.e.\ allowing the same samples to appear in multiple tasks. This makes the CI misleading in that it takes into account the randomness of the sampler but not the data itself. To quantify the extent of this problem, we conduct a comparative analysis between CIs computed with and without replacement. These reveal a notable underestimation by the predominant method. This observation calls for a reevaluation of how we interpret confidence intervals and the resulting conclusions in FSL comparative studies. Our research demonstrates that the use of paired tests can partially address this issue. Additionally, we explore methods to further reduce the (size of the) CI by strategically sampling tasks of a specific size. We also introduce a new optimized benchmark, which can be accessed at https://github.com/RafLaf/FSL-benchmark-again
Code (1)
Tasks
Few-Shot LearningSimilar Papers 제목 키워드 기반
Simultaneous Inference for Local Structural Parameters with Random Forests
We construct simultaneous confidence intervals for solutions to conditional moment equations. The intervals are built around a class of nonparametric regression algorithms based on subsampled kernels. This class encompas…
Causal InferenceregressionEliciting Kemeny Rankings
We formulate the problem of eliciting agents' preferences with the goal of finding a Kemeny ranking as a Dueling Bandits problem. Here the bandits' arms correspond to alternatives that need to be ranked and the feedback …
Unmatched Does Not Mean False: Incomplete Reference Sets Can Reverse Calibration Rankings in Open-Ended Theory-of-Mind Tracking
Open-ended Theory-of-Mind (ToM) trackers emit valid beliefs absent from finite references. A finite-reference-plus-matcher pipeline marks unmatched outputs false, creating proxy labels that can reverse proper-score model…
High-Dimensional Confidence Regions in Sparse MRI
One of the most promising solutions for uncertainty quantification in high-dimensional statistics is the debiased LASSO that relies on unconstrained $\ell_1$-minimization. The initial works focused on real Gaussian desig…
Compressive SensingUncertainty QuantificationA probabilistic estimation and prediction technique for dynamic continuous social science models: The evolution of the attitude of the Basque Country population towards ETA as a case study
In this paper, we present a computational technique to deal with uncertainty in dynamic continuous models in Social Sciences. Considering data from surveys, the method consists of determining the probability distribution…
Survey