Efficient Sparse Training with Structured Dropout

2024-11-02 · Andy Lo

Dropout is a common regularisation technique in deep learning that improves generalisation. Even though it introduces sparsity and thus potential for higher throughput, it usually cannot bring speed-ups on GPUs due to its unstructured nature. In this project, I experiment with SparseDrop, a structured, hardware-friendly variant of dropout that can exploit such sparsity. I provide a CUDA implementation of SparseDrop, achieving speed-ups against its dense counterpart even at low sparsity levels. The empirical results demonstrate that SparseDrop provides similar, or sometimes even better, regularisation properties as standard dropout. This suggests its potential as a drop-in replacement to standard dropout with faster training speeds. The source code is available at https://github.com/andylolu2/sparse-dropout

📄 PDF Abstract BibTeX arXiv:2411.01238

Code (1)

andylolu2/sparse-dropout 공식 구현 pytorch

Methods 이 논문이 사용한 방법론

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

ASNI: Adaptive Structured Noise Injection for shallow and deep neural networks

2019-09-21 · Beyrem Khalfaoui, Joseph Boyd, Jean-Philippe Vert

Dropout is a regularisation technique in neural network training where unit activations are randomly set to zero with a given probability \emph{independently}. In this work, we propose a generalisation of dropout and oth…

Structured Bayesian Pruning via Log-Normal Multiplicative Noise

2017-05-20 · NeurIPS 2017 12 · Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, Dmitry Vetrov

Dropout-based regularization methods can be regarded as injecting random noise with pre-defined magnitude to different parts of the neural network during training. It was recently shown that Bayesian dropout procedure no…

Revisiting Structured Dropout

2022-10-05 · Yiren Zhao, Oluwatomisin Dada, Xitong Gao, Robert D Mullins

Large neural networks are often overparameterised and prone to overfitting, Dropout is a widely used regularization technique to combat overfitting and improve model generalization. However, unstructured Dropout is not a…

Scheduling

Spatially-Coupled Neural Network Architectures

2019-07-03 · Arman Hasanzadeh, Nagaraj T. Janakiraman, Vamsi K. Amalladinne, Krishna R. Narayanan

In this work, we leverage advances in sparse coding techniques to reduce the number of trainable parameters in a fully connected neural network. While most of the works in literature impose $\ell_1$ regularization, DropO…

Feature Importance

PairDropGS: Paired Dropout-Induced Consistency Regularization for Sparse-View Gaussian Splatting

2026-05-12 · Hantang Li, Qiang Zhu, Xiandong Meng, Xingtao Wang 외 arxiv

Dropout-based sparse-view 3D Gaussian Splatting (3DGS) methods alleviate overfitting by randomly suppressing Gaussian primitives during training. Existing methods mainly focus on designing increasingly sophisticated drop…

Representation Learning