Dropout Inference with Non-Uniform Weight Scaling
Dropout as regularization has been used extensively to prevent overfitting for training neural networks. During training, units and their connections are randomly dropped, which could be considered as sampling many different submodels from the original model. At test time, weight scaling and Monte Carlo approximation are two widely applied approaches to approximate the outputs. Both approaches work well practically when all submodels are low-bias complex learners. However, in this work, we demonstrate scenarios where some submodels behave closer to high-bias models and a non-uniform weight scaling is a better approximation for inference.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Variational Bayesian Dropout with a Hierarchical Prior
Variational dropout (VD) is a generalization of Gaussian dropout, which aims at inferring the posterior of network weights based on a log-uniform prior on them to learn these weights as well as dropout rate simultaneousl…
Well-calibrated Model Uncertainty with Temperature Scaling for Dropout Variational Inference
Model uncertainty obtained by variational Bayesian inference with Monte Carlo dropout is prone to miscalibration. The uncertainty does not represent the model error well. In this paper, temperature scaling is extended to…
Bayesian InferenceVariational InferenceCalibration of Model Uncertainty for Dropout Variational Inference
The model uncertainty obtained by variational Bayesian inference with Monte Carlo dropout is prone to miscalibration. In this paper, different logit scaling methods are extended to dropout variational inference to recali…
Bayesian InferenceVariational InferenceSurprising properties of dropout in deep networks
We analyze dropout in deep networks with rectified linear units and the quadratic loss. Our results expose surprising differences between the behavior of dropout and more traditional regularizers like weight decay. For e…
An empirical analysis of dropout in piecewise linear networks
The recently introduced dropout training criterion for neural networks has been the subject of much attention due to its simplicity and remarkable effectiveness as a regularizer, as well as its interpretation as a traini…