Mitigating deep double descent by concatenating inputs
The double descent curve is one of the most intriguing properties of deep neural networks. It contrasts the classical bias-variance curve with the behavior of modern neural networks, occurring where the number of samples nears the number of parameters. In this work, we explore the connection between the double descent phenomena and the number of samples in the deep neural network setting. In particular, we propose a construction which augments the existing dataset by artificially increasing the number of samples. This construction empirically mitigates the double descent curve in this setting. We reproduce existing work on deep double descent, and observe a smooth descent into the overparameterized region for our construction. This occurs both with respect to the model size, and with respect to the number epochs.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Multiple Descent in the Multiple Random Feature Model
Recent works have demonstrated a double descent phenomenon in over-parameterized learning. Although this phenomenon has been investigated by recent works, it has not been fully understood in theory. In this paper, we inv…
Ensemble LearningmodelregressionLoRA-GGPO: Mitigating Double Descent in LoRA Fine-Tuning via Gradient-Guided Perturbation Optimization
Large Language Models (LLMs) have achieved remarkable success in natural language processing, but their full fine-tuning remains resource-intensive. Parameter-Efficient Fine-Tuning (PEFT) methods, such as Low-Rank Adapta…
Natural Language Understandingparameter-efficient fine-tuningTraining Data Size Induced Double Descent For Denoising Neural Networks and the Role of Training Noise Level
When training a denoising neural network, we show that more data isn’t more beneficial. In fact the generalization error versus number of of training data points is a double descent curve. Training a network to denoise n…
DenoisingMultiple Descents in Unsupervised Learning: The Role of Noise, Domain Shift and Anomalies
The phenomenon of double descent has recently gained attention in supervised learning. It challenges the conventional wisdom of the bias-variance trade-off by showcasing a surprising behavior. As the complexity of the mo…
Deep Double Descent for Time Series Forecasting: Avoiding Undertrained Models
Deep learning models, particularly Transformers, have achieved impressive results in various domains, including time series forecasting. While existing time series literature primarily focuses on model architecture modif…
Data AugmentationTime SeriesTime Series Forecasting