Dropout: Explicit Forms and Capacity Control
We investigate the capacity control provided by dropout in various machine learning problems. First, we study dropout for matrix completion, where it induces a data-dependent regularizer that, in expectation, equals the weighted trace-norm of the product of the factors. In deep learning, we show that the data-dependent regularizer due to dropout directly controls the Rademacher complexity of the underlying class of deep neural networks. These developments enable us to give concrete generalization error bounds for the dropout algorithm in both matrix completion as well as training deep neural networks. We evaluate our theoretical findings on real-world datasets, including MovieLens, MNIST, and Fashion-MNIST.
Code (0)
등록된 구현이 없습니다.
Tasks
BIG-bench Machine LearningMatrix CompletionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Do deep nets really need weight decay and dropout?
The impressive success of modern deep neural networks on computer vision tasks has been achieved through models of very large capacity compared to the number of available training examples. This overparameterization is o…
Data AugmentationObject RecognitionExplicit Dropout: Deterministic Regularization for Transformer Architectures
Dropout is a widely used regularization technique in deep learning, but its effects are typically realized through stochastic masking rather than explicit optimization objectives. We propose a deterministic formulation t…
Audio ClassificationImage ClassificationAction DetectionThe Implicit and Explicit Regularization Effects of Dropout
Dropout is a widely-used regularization technique, often required to obtain state-of-the-art for a number of architectures. This work demonstrates that dropout introduces two distinct but entangled regularization effects…
Model-Free Learning and Optimal Policy Design in Multi-Agent MDPs Under Probabilistic Agent Dropout
This work studies a multi-agent Markov decision process (MDP) that can undergo agent dropout and the computation of policies for the post-dropout system based on control and sampling of the pre-dropout system. The centra…
Sparseout: Controlling Sparsity in Deep Networks
Dropout is commonly used to help reduce overfitting in deep neural networks. Sparsity is a potentially important property of neural networks, but is not explicitly controlled by Dropout-based regularization. In this work…
General Classificationimage-classificationImage ClassificationLanguage Modelling