Drop Dropout on Single-Epoch Language Model Pretraining

2025-05-30 · Houjun Liu, John Bauer, Christopher D. Manning

Originally, dropout was seen as a breakthrough regularization technique that reduced overfitting and improved performance in almost all applications of deep learning by reducing overfitting. Yet, single-epoch pretraining tasks common to modern LLMs yield minimal overfitting, leading to dropout not being used for large LLMs. Nevertheless, no thorough empirical investigation has been done on the role of dropout in LM pretraining. Through experiments in single-epoch pretraining of both masked (BERT) and autoregressive (Pythia 160M and 1.4B) LMs with varying levels of dropout, we find that downstream performance in language modeling, morpho-syntax (BLiMP), question answering (SQuAD), and natural-language inference (MNLI) improves when dropout is not applied during pretraining. We additionally find that the recently-introduced "early dropout" also degrades performance over applying no dropout at all. We further investigate the models' editability, and find that models trained without dropout are more successful in gradient-based model editing (MEND) and equivalent in representation-based model editing (ReFT). Therefore, we advocate to drop dropout during single-epoch pretraining.

📄 PDF Abstract BibTeX arXiv:2505.24788

Code (1)

jemoka/dropfree 공식 구현 pytorch

Tasks

Language ModelingLanguage ModellingModel EditingNatural Language Inference Question Answering

Methods 이 논문이 사용한 방법론

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Adaptive Tabu Dropout for Regularization of Deep Neural Network

2024-12-31 · Md. Tarek Hasan, Arifa Akter, Mohammad Nazmush Shamael, Md Al Emran Hossain 외

Dropout is an effective strategy for the regularization of deep neural networks. Applying tabu to the units that have been dropped in the recent epoch and retaining them for training ensures diversification in dropout. I…

Pretraining Language Models with Subword Regularization: An Empirical Study of BPE Dropout in Low-Resource NLP

2026-05-13 · Ruan Visser, Trienko Grobler, Marcel Dunaiski arxiv

Subword regularization methods such as BPE dropout are typically applied only during fine-tuning, while pretraining is usually done with deterministic tokenization. This creates a potential segmentation mismatch between …

Dropout-GAN: Learning from a Dynamic Ensemble of Discriminators

2018-07-30 · Gonçalo Mordido, Haojin Yang, Christoph Meinel

We propose to incorporate adversarial dropout in generative multi-adversarial networks, by omitting or dropping out, the feedback of each discriminator in the framework with some probability at the end of each batch. Our…

Diversity

Enhancing Transformer Training Efficiency with Dynamic Dropout

2024-11-05 · Hanrui Yan, Dan Shao

We introduce Dynamic Dropout, a novel regularization technique designed to enhance the training efficiency of Transformer models by dynamically adjusting the dropout rate based on training epochs or validation loss impro…

Convergence, design and training of continuous-time dropout as a random batch method

2025-10-15 · Antonio Álvarez-López, Martín Hernández arxiv

We study dropout regularization in continuous-time models through the lens of random-batch methods -- a family of stochastic sampling schemes originally devised to reduce the computational cost of interacting particle sy…