Parallel Dither and Dropout for Regularising Deep Neural Networks
Effective regularisation during training can mean the difference between success and failure for deep neural networks. Recently, dither has been suggested as alternative to dropout for regularisation during batch-averaged stochastic gradient descent (SGD). In this article, we show that these methods fail without batch averaging and we introduce a new, parallel regularisation method that may be used without batch averaging. Our results for parallel-regularised non-batch-SGD are substantially better than what is possible with batch-SGD. Furthermore, our results demonstrate that dither and dropout are complimentary.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Dither is Better than Dropout for Regularising Deep Neural Networks
Regularisation of deep neural networks (DNN) during training is critical to performance. By far the most popular method is known as dropout. Here, cast through the prism of signal processing theory, we compare and contra…
Regularising Deep Networks with Deep Generative Models
We develop a new method for regularising neural networks. We learn a probability distribution over the activations of all layers of the model and then insert imputed values into the network during training. We obtain a p…
Data AugmentationImputationTaming the ReLU with Parallel Dither in a Deep Neural Network
Rectified Linear Units (ReLU) seem to have displaced traditional 'smooth' nonlinearities as activation-function-du-jour in many - but not all - deep neural network (DNN) applications. However, nobody seems to know why. I…
Distortion-Controlled Dithering with Reduced Recompression Rate
Dithering is a technique that can improve human perception of low-resolution data by reducing quantization artifacts. In this work we formalize and analytically justify two metrics for quantization artifact prominence, u…
Data CompressionImage CompressionQuantization1-bit Localization Scheme for Radar using Dithered Quantized Compressed Sensing
We present a novel scheme allowing for 2D target localization using highly quantized 1-bit measurements from a Frequency Modulated Continuous Wave (FMCW) radar with two receiving antennas. Quantization of radar signals i…
compressed sensingQuantization