The Double Descent Behavior in Two Layer Neural Network for Binary Classification
Recent studies observed a surprising concept on model test error called the double descent phenomenon, where the increasing model complexity decreases the test error first and then the error increases and decreases again. To observe this, we work on a two layer neural network model with a ReLU activation function designed for binary classification under supervised learning. Our aim is to observe and investigate the mathematical theory behind the double descent behavior of model test error for varying model sizes. We quantify the model size by the ratio of number of training samples to the dimension of the model. Due to the complexity of the empirical risk minimization procedure, we use the Convex Gaussian Min Max Theorem to find a suitable candidate for the global training loss.
Code (0)
등록된 구현이 없습니다.
Tasks
Binary ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Regularization-wise double descent: Why it occurs and how to eliminate it
The risk of overparameterized models, in particular deep neural networks, is often double-descent shaped as a function of the model size. Recently, it was shown that the risk as a function of the early-stopping time can …
On double-descent in uncertainty quantification in overparametrized models
Uncertainty quantification is a central challenge in reliable and trustworthy machine learning. Naive measures such as last-layer scores are well-known to yield overconfident estimates in the context of overparametrized …
Binary ClassificationUncertainty QuantificationFrozen Overparameterization: A Double Descent Perspective on Transfer Learning of Deep Neural Networks
We study the generalization behavior of transfer learning of deep neural networks (DNNs). We adopt the overparameterization perspective -- featuring interpolation of the training data (i.e., approximately zero train erro…
image-classificationImage ClassificationTransfer LearningEarly Stopping in Deep Networks: Double Descent and How to Eliminate it
Over-parameterized models, such as large deep networks, often exhibit a double descent phenomenon, whereas a function of model size, error first decreases, increases, and decreases at last. This intriguing double descent…
VC Theoretical Explanation of Double Descent
There has been growing interest in generalization performance of large multilayer neural networks that can be trained to achieve zero training error, while generalizing well on test data. This regime is known as 'second …
Generalization Bounds