Manipulating Sparse Double Descent
This paper investigates the double descent phenomenon in two-layer neural networks, focusing on the role of L1 regularization and representation dimensions. It explores an alternative double descent phenomenon, named sparse double descent. The study emphasizes the complex relationship between model complexity, sparsity, and generalization, and suggests further research into more diverse models and datasets. The findings contribute to a deeper understanding of neural network training and optimization.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Sparse Double Descent: Where Network Pruning Aggravates Overfitting
People usually believe that network pruning not only reduces the computational cost of deep networks, but also prevents overfitting by decreasing model capacity. However, our work surprisingly discovers that network prun…
Network PruningSparse Double Descent in Vision Transformers: real or phantom threat?
Vision transformers (ViT) have been of broad interest in recent theoretical and empirical works. They are state-of-the-art thanks to their attention-based approach, which boosts the identification of key features and pat…
Inductive BiasOverfitting Can Be Harmless for Basis Pursuit, But Only to a Degree
Recently, there have been significant interests in studying the so-called "double-descent" of the generalization error of linear regression models under the overparameterized and overfitting regime, with the hope that su…
compressed sensingregressionThe Quest of Finding the Antidote to Sparse Double Descent
In energy-efficient schemes, finding the optimal size of deep learning models is very important and has a broad impact. Meanwhile, recent studies have reported an unexpected phenomenon, the sparse double descent: as the …
image-classificationImage ClassificationClass-wise Activation Unravelling the Engima of Deep Double Descent
Double descent presents a counter-intuitive aspect within the machine learning domain, and researchers have observed its manifestation in various models and tasks. While some theoretical explanations have been proposed f…