On double-descent in uncertainty quantification in overparametrized models
Uncertainty quantification is a central challenge in reliable and trustworthy machine learning. Naive measures such as last-layer scores are well-known to yield overconfident estimates in the context of overparametrized neural networks. Several methods, ranging from temperature scaling to different Bayesian treatments of neural networks, have been proposed to mitigate overconfidence, most often supported by the numerical observation that they yield better calibrated uncertainty measures. In this work, we provide a sharp comparison between popular uncertainty measures for binary classification in a mathematically tractable model for overparametrized neural networks: the random features model. We discuss a trade-off between classification accuracy and calibration, unveiling a double descent like behavior in the calibration curve of optimally regularized estimators as a function of overparametrization. This is in contrast with the empirical Bayes method, which we show to be well calibrated in our setting despite the higher generalization error and overparametrization.
Code (1)
Tasks
Binary ClassificationUncertainty QuantificationSimilar Papers 제목 키워드 기반
Analysis of Interpolating Regression Models and the Double Descent Phenomenon
A regression model with more parameters than data points in the training data is overparametrized and has the capability to interpolate the training data. Based on the classical bias-variance tradeoff expressions, it is …
regressionVariational Deep Learning via Implicit Regularization
Modern deep learning models generalize remarkably well in-distribution, despite being overparametrized and trained with little to no explicit regularization. Instead, current theory credits implicit regularization impose…
Deep LearningInductive BiasSequential Decision MakingUncertainty Quantification+1Double descent for least-squares interpolation on contaminated data: A simulation study
Overparametrized models can exhibit an excellent generalization performance, although they should be prone to overfitting according to classical statistical theory. The discovery of the "double descent", indicating that …
Asymptotic normality and confidence intervals for derivatives of 2-layers neural network in the random features model
This paper studies two-layers Neural Networks (NN), where the first layer contains random weights, and the second layer is trained using Ridge regularization. This model has been the focus of numerous recent works, showi…
regressionUncertainty QuantificationA dynamic view of the double descent
It has been observed by Belkin et al.\ that overparametrized neural networks exhibit a `double descent' phenomenon. That is, as the model complexity, as reflected in the number of features, increases, the training error …