Regularizing Activation Distribution for Training Binarized Deep Networks
Binarized Neural Networks (BNNs) can significantly reduce the inference latency and energy consumption in resource-constrained devices due to their pure-logical computation and fewer memory accesses. However, training BNNs is difficult since the activation flow encounters degeneration, saturation, and gradient mismatch problems. Prior work alleviates these issues by increasing activation bits and adding floating-point scaling factors, thereby sacrificing BNN's energy efficiency. In this paper, we propose to use distribution loss to explicitly regularize the activation flow, and develop a framework to systematically formulate the loss. Our experiments show that the distribution loss can consistently improve the accuracy of BNNs without losing their energy benefits. Moreover, equipped with the proposed regularization, BNN training is shown to be robust to the selection of hyper-parameters including optimizer and learning rate.
Code (1)
Similar Papers 제목 키워드 기반
Binarized High-Efficiency RAW Video Restoration and Beyond
RAW video restoration is fundamental to high-quality low-level perception and serves as the basis for a wide range of downstream vision applications. While binary neural networks (BNNs) enable efficient lightweight deplo…
Monocular Depth EstimationVideo RestorationImage EnhancementObject DetectionFast Binarized Neural Network Training with Partial Pre-training
Binarized neural networks, networks with weights and activations constrained to lie in a 2-element set, allow for more time- and resource-efficient inference than standard floating-point networks. However, binarized neur…
Binarized Diffusion Model for Image Super-Resolution
Advanced diffusion models (DMs) perform impressively in image super-resolution (SR), but the high memory and computational costs hinder their deployment. Binarization, an ultra-compression algorithm, offers the potential…
AttributeBinarizationImage Super-Resolutionmodel+2On the discriminability of samples in the Hamming space of binarized ReLU activations
Binarized ReLU activations are considered as a metric space equipped with the Hamming distance. While for two layer ReLU networks with random Gaussian weights it can be shown theoretically that local metric properties a…
Binarized Spectral Compressive Imaging
Existing deep learning models for hyperspectral image (HSI) reconstruction achieve good performance but require powerful hardwares with enormous memory and computational resources. Consequently, these methods can hardly …
Binarization