On the Asymptotic Learning Curves of Kernel Ridge Regression under Power-law Decay
The widely observed 'benign overfitting phenomenon' in the neural network literature raises the challenge to the 'bias-variance trade-off' doctrine in the statistical learning theory. Since the generalization ability of the 'lazy trained' over-parametrized neural network can be well approximated by that of the neural tangent kernel regression, the curve of the excess risk (namely, the learning curve) of kernel ridge regression attracts increasing attention recently. However, most recent arguments on the learning curve are heuristic and are based on the 'Gaussian design' assumption. In this paper, under mild and more realistic assumptions, we rigorously provide a full characterization of the learning curve: elaborating the effect and the interplay of the choice of the regularization parameter, the source condition and the noise. In particular, our results suggest that the 'benign overfitting phenomenon' exists in very wide neural networks only when the noise level is small.
Code (0)
등록된 구현이 없습니다.
Tasks
Learning TheoryregressionSimilar Papers 제목 키워드 기반
Learning curves for Gaussian process regression with power-law priors and targets
We characterize the power-law asymptotics of learning curves for Gaussian process regression (GPR) under the assumption that the eigenspectrum of the prior and the eigenexpansion coefficients of the target function follo…
GPRregressionSharp Asymptotics of Kernel Ridge Regression Beyond the Linear Regime
The generalization performance of kernel ridge regression (KRR) exhibits a multi-phased pattern that crucially depends on the scaling relationship between the sample size $n$ and the underlying dimension $d$. This phenom…
regressionPrecise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regression
As modern machine learning models continue to advance the computational frontier, it has become increasingly important to develop precise estimates for expected performance improvements under different model and data sca…
regressionSequential Kernel Embedding for Mediated and Time-Varying Dose Response Curves
We propose simple nonparametric estimators for mediated and time-varying dose response curves based on kernel ridge regression. By embedding Pearl's mediation formula and Robins' g-formula with kernels, we allow treatmen…
Causal InferencecounterfactualLearning between the peaks: sharp asymptotics for kernel ridge regression under power-law anisotropy
We study kernel ridge regression under anisotropic Gaussian data, where the input covariance decays as a power law with exponent $α\geq 0$ for polynomial inner-product kernels. We derive asymptotically sharp expressions …