Backdrop: Stochastic Backpropagation

2018-06-04 · ICLR 2019 5 · Siavash Golkar, Kyle Cranmer

We introduce backdrop, a flexible and simple-to-implement method, intuitively described as dropout acting only along the backpropagation pipeline. Backdrop is implemented via one or more masking layers which are inserted at specific points along the network. Each backdrop masking layer acts as the identity in the forward pass, but randomly masks parts of the backward gradient propagation. Intuitively, inserting a backdrop layer after any convolutional layer leads to stochastic gradients corresponding to features of that scale. Therefore, backdrop is well suited for problems in which the data have a multi-scale, hierarchical structure. Backdrop can also be applied to problems with non-decomposable loss functions where standard SGD methods are not well suited. We perform a number of experiments and demonstrate that backdrop leads to significant improvements in generalization.

📄 PDF Abstract BibTeX arXiv:1806.01337

Code (1)

dexgen/backdrop 공식 구현 pytorch

Methods 이 논문이 사용한 방법론

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

SGD Stochastic Gradient Descent is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using…

Backprop-Q: Generalized Backpropagation for Stochastic Computation Graphs

2018-07-25 · Xiaoran Xu, Songpeng Zu, Yuan Zhang, Hanning Zhou 외

In real-world scenarios, it is appealing to learn a model carrying out stochastic operations internally, known as stochastic computation graphs (SCGs), rather than learning a deterministic mapping. However, standard back…

Reinforcement Learning Reinforcement Learning (RL)

MuProp: Unbiased Backpropagation for Stochastic Neural Networks

2015-11-16 · Shixiang Gu, Sergey Levine, Ilya Sutskever, andriy mnih

Deep neural networks are powerful parametric models that can be trained efficiently using the backpropagation algorithm. Stochastic neural networks combine the power of large parametric functions with that of graphical m…

Generalized Stochastic Backpropagation

2021-01-01 · Amine Echraibi, Joachim Flocon Cholet, Stéphane Gosselin, Sandrine Vaton

Backpropagating gradients through random variables is at the heart of numerous machine learning applications. In this paper, we present a general framework for deriving stochastic backpropagation rules for any distributi…

An In-depth Study of Stochastic Backpropagation

2022-09-30 · Jun Fang, Mingze Xu, Hao Chen, Bing Shuai 외

In this paper, we provide an in-depth study of Stochastic Backpropagation (SBP) when training deep neural networks for standard image classification and object detection tasks. During backward propagation, SBP calculates…

GPU image-classification Image Classification object-detection+1

Fast Second-Order Stochastic Backpropagation for Variational Inference

2015-09-09 · Kai Fan, Ziteng Wang, Jeff Beck, James Kwok 외

We propose a second-order (Hessian or Hessian-free) based optimization method for variational inference inspired by Gaussian backpropagation, and argue that quasi-Newton optimization can be developed as well. This is acc…

regression Variational Inference