Dropout Training as Adaptive Regularization
Dropout and other feature noising schemes control overfitting by artificially corrupting the training data. For generalized linear models, dropout performs a form of adaptive regularization. Using this viewpoint, we show that the dropout regularizer is first-order equivalent to an L2 regularizer applied after scaling the features by an estimate of the inverse diagonal Fisher information matrix. We also establish a connection to AdaGrad, an online learning algorithm, and find that a close relative of AdaGrad operates by repeatedly solving linear dropout-regularized problems. By casting dropout as regularization, we develop a natural semi-supervised algorithm that uses unlabeled data to create a better adaptive regularizer. We apply this idea to document classification tasks, and show that it consistently boosts the performance of dropout training, improving on state-of-the-art results on the IMDB reviews dataset.
Code (0)
등록된 구현이 없습니다.
Tasks
Document ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Adaptive Tabu Dropout for Regularization of Deep Neural Network
Dropout is an effective strategy for the regularization of deep neural networks. Applying tabu to the units that have been dropped in the recent epoch and retaining them for training ensures diversification in dropout. I…
The Flip Side of the Reweighted Coin: Duality of Adaptive Dropout and Regularization
Among the most successful methods for sparsifying deep (neural) networks are those that adaptively mask the network weights throughout training. By examining this masking, or dropout, in the linear case, we uncover a dua…
Adaptive Dropout: Unleashing Dropout across Layers for Generalizable Image Super-Resolution
Blind Super-Resolution(blind SR) aims to enhance the model's generalization ability with unknown degradation, yet it still encounters severe overfitting issues. Some previous methods inspired by dropout, which enhanc…
AttributeBlind Super-ResolutionImage RestorationImage Super-Resolution+1Lyapunov-Based Dropout Deep Neural Network (Lb-DDNN) Controller
Deep neural network (DNN)-based adaptive controllers can be used to compensate for unstructured uncertainties in nonlinear dynamic systems. However, DNNs are also very susceptible to overfitting and co-adaptation. Dropou…
Dropout Training, Data-dependent Regularization, and Generalization Bounds
We study the problem of generalization guarantees for dropout training. A general framework is first proposed for learning procedures with random perturbation on model parameters. The generalization error is bounded…
Generalization Bounds