Explicit Regularisation in Gaussian Noise Injections
We study the regularisation induced in neural networks by Gaussian noise injections (GNIs). Though such injections have been extensively studied when applied to data, there have been few studies on understanding the regularising effect they induce when applied to network activations. Here we derive the explicit regulariser of GNIs, obtained by marginalising out the injected noise, and show that it penalises functions with high-frequency components in the Fourier domain; particularly in layers closer to a neural network's output. We show analytically and empirically that such regularisation produces calibrated classifiers with large classification margins.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections
Gaussian noise injections (GNIs) are a family of simple and widely-used regularisation methods for training neural networks, where one injects additive or multiplicative Gaussian noise to the network activations at every…
Differential Privacy of Gaussian Process Posterior Sampling
We study the privacy of releasing posterior sample paths from a Gaussian process (GP) when the entire training set including covariates and responses is private. Unlike standard differential-privacy (DP) mechanisms that …
On the Inherent Regularization Effects of Noise Injection During Training
Randomly perturbing networks during the training process is a commonly used approach to improving generalization performance. In this paper, we present a theoretical study of one particular way of random perturbation, wh…
Patch-based learning of adaptive Total Variation parameter maps for blind image denoising
We consider a patch-based learning approach defined in terms of neural networks to estimate spatially adaptive regularisation parameter maps for image denoising with weighted Total Variation (TV) and test it to situation…
Binary ClassificationDenoisingImage DenoisingModel Selection+1Enhanced Denoising and Convergent Regularisation Using Tweedie Scaling
The inherent ill-posed nature of image reconstruction problems, due to limitations in the physical acquisition process, is typically addressed by introducing a regularisation term that incorporates prior knowledge about …
DenoisingImage Reconstruction