Floating-Point Quantization Analysis of Multi-Layer Perceptron Artificial Neural Networks
The impact of quantization in Multi-Layer Perceptron (MLP) Artificial Neural Networks (ANNs) is presented in this paper. In this architecture, the constant increase in size and the demand to decrease bit precision are two factors that contribute to the significant enlargement of quantization errors. We introduce an analytical tool that models the propagation of Quantization Noise Power (QNP) in floating-point MLP ANNs. Contrary to the state-of-the-art approach, which compares the exact and quantized data experimentally, the proposed algorithm can predict the QNP theoretically when the effect of operation quantization and Coefficient Quantization Error (CQE) are considered. This supports decisions in determining the required precision during the hardware design. The algorithm is flexible in handling MLP ANNs of user-defined parameters, such as size and type of activation function. Additionally, a simulation environment is built that can perform each operation on an adjustable bit precision. The accuracy of the QNP calculation is verified with two publicly available benchmarked datasets, using the default precision simulation environment as a reference.
Code (1)
Tasks
QuantizationSimilar Papers 제목 키워드 기반
Phoenix: A Low-Precision Floating-Point Quantization Oriented Architecture for Convolutional Neural Networks
Convolutional neural networks (CNNs) achieve state-of-the-art performance at the cost of becoming deeper and larger. Although quantization (both fixed-point and floating-point) has proven effective for reducing storage a…
QuantizationF-BFQ: Flexible Block Floating-Point Quantization Accelerator for LLMs
Large Language Models (LLMs) have become increasingly prominent for daily tasks, from improving sound-totext translation to generating additional frames for the latest video games. With the help of LLM inference framewor…
Optimal Quantization for Batch Normalization in Neural Network Deployments and Beyond
Quantized Neural Networks (QNNs) use low bit-width fixed-point numbers for representing weight parameters and activations, and are often used in real-world applications due to their saving of computation resources and re…
QuantizationDesign and Analysis of Uplink and Downlink Communications for Federated Learning
Communication has been known to be one of the primary bottlenecks of federated learning (FL), and yet existing studies have not addressed the efficient communication design, particularly in wireless FL where both uplink …
Federated LearningQuantizationIFQ-Net: Integrated Fixed-point Quantization Networks for Embedded Vision
Deploying deep models on embedded devices has been a challenging problem since the great success of deep learning based networks. Fixed-point networks, which represent their data with low bits fixed-point and thus give r…
Face DetectionImage ClassificationQuantization