Bitwidth-Specific Logarithmic Arithmetic for Future Hardware-Accelerated Training
While advancements in quantization have significantly reduced the computational costs of inference in deep learning, training still predominantly relies on complex floating-point arithmetic. Low-precision fixed-point training presents a compelling alternative. This work introduces a novel enhancement in low-precision logarithmic fixed-point training, geared towards future hardware accelerator designs. We propose incorporating bitwidth in the design of approximations to arithmetic operations. To this end, we introduce a new hardware-friendly, piece-wise linear approximation for logarithmic addition. Using simulated annealing, we optimize this approximation at different precision levels. A C++ bit-true simulation demonstrates training of VGG-11 and VGG-16 models on CIFAR-100 and TinyImageNet, respectively, using 12-bit integer arithmetic with minimal accuracy degradation compared to 32-bit floating-point training. Our hardware study reveals up to 32.5% reduction in area and 53.5% reduction in energy consumption for the proposed LNS multiply-accumulate units compared to that of linear fixed-point equivalents.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Heterogeneous Bitwidth Binarization in Convolutional Neural Networks
Recent work has shown that fast, compact low-bitwidth neural networks can be surprisingly accurate. These networks use homogeneous binarization: all parameters in each layer or (more commonly) the whole model have the sa…
BinarizationSDQ: Stochastic Differentiable Quantization with Mixed Precision
In order to deploy deep models in a computationally efficient manner, model quantization approaches have been frequently used. In addition, as new hardware that supports mixed bitwidth arithmetic operations, recent resea…
Knowledge DistillationNeural Architecture SearchQuantizationAMED: Automatic Mixed-Precision Quantization for Edge Devices
Quantized neural networks are well known for reducing the latency, power consumption, and model size without significant harm to the performance. This makes them highly appropriate for systems with limited resources and …
QuantizationLow-Bitwidth Floating Point Quantization for Efficient High-Quality Diffusion Models
Diffusion models are emerging models that generate images by iteratively denoising random Gaussian noise using deep neural networks. These models typically exhibit high computational and memory demands, necessitating eff…
DenoisingQuantizationFully Quantized Network for Object Detection
Efficient neural network inference is important in a number of practical domains, such as deployment in mobile settings. An effective method for increasing inference efficiency is to use low bitwidth arithmetic, which c…
Efficient Neural NetworkObjectobject-detectionObject Detection+1