Overcoming Challenges in Fixed Point Training of Deep Convolutional Networks
It is known that training deep neural networks, in particular, deep convolutional networks, with aggressively reduced numerical precision is challenging. The stochastic gradient descent algorithm becomes unstable in the presence of noisy gradient updates resulting from arithmetic with limited numeric precision. One of the well-accepted solutions facilitating the training of low precision fixed point networks is stochastic rounding. However, to the best of our knowledge, the source of the instability in training neural networks with noisy gradient updates has not been well investigated. This work is an attempt to draw a theoretical connection between low numerical precision and training algorithm stability. In doing so, we will also propose and verify through experiments methods that are able to improve the training performance of deep convolutional networks in fixed point.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Phoenix: A Low-Precision Floating-Point Quantization Oriented Architecture for Convolutional Neural Networks
Convolutional neural networks (CNNs) achieve state-of-the-art performance at the cost of becoming deeper and larger. Although quantization (both fixed-point and floating-point) has proven effective for reducing storage a…
QuantizationOvercoming Distribution Shifts in Plug-and-Play Methods with Test-Time Training
Plug-and-Play Priors (PnP) is a well-known class of methods for solving inverse problems in computational imaging. PnP methods combine physical forward models with learned prior models specified as image denoisers. A com…
Image ReconstructionUnderstanding and Overcoming the Challenges of Efficient Transformer Quantization
Transformer-based architectures have become the de-facto standard models for a wide range of Natural Language Processing tasks. However, their memory footprint and high latency are prohibitive for efficient deployment an…
QuantizationFixed-point optimization of deep neural networks with adaptive step size retraining
Fixed-point optimization of deep neural networks plays an important role in hardware based design and low-power implementations. Many deep neural networks show fairly good performance even with 2- or 3-bit precision when…
QuantizationSVD-DIP: Overcoming the Overfitting Problem in DIP-based CT Reconstruction
The deep image prior (DIP) is a well-established unsupervised deep learning method for image reconstruction; yet it is far from being flawless. The DIP overfits to noise if not early stopped, or optimized via a regulariz…
CT ReconstructionImage Reconstruction