Hardware-oriented Approximation of Convolutional Neural Networks

2016-04-11 · Philipp Gysel, Mohammad Motamedi, Soheil Ghiasi

High computational complexity hinders the widespread usage of Convolutional Neural Networks (CNNs), especially in mobile devices. Hardware accelerators are arguably the most promising approach for reducing both execution time and power consumption. One of the most important steps in accelerator development is hardware-oriented model approximation. In this paper we present Ristretto, a model approximation framework that analyzes a given CNN with respect to numerical resolution used in representing weights and outputs of convolutional and fully connected layers. Ristretto can condense models by using fixed point arithmetic and representation instead of floating point. Moreover, Ristretto fine-tunes the resulting fixed point network. Given a maximum error tolerance of 1%, Ristretto can successfully condense CaffeNet and SqueezeNet to 8-bit. The code for Ristretto is available.

📄 PDF Abstract BibTeX arXiv:1604.03168

Code (1)

pmgysel/caffe 공식 구현

Methods 이 논문이 사용한 방법론

ReLU How Do I Communicate to Expedia? How Do I Communicate to Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Live Support & Special Travel…

Residual Connection 설명 없음

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Average Pooling 설명 없음

Fire Module 설명 없음

Global Average Pooling Global Average Pooling is a pooling operation designed to replace fully connected layers in classical CNNs. The idea is to generate one feature map for each corresponding…

1x1 Convolution A 1 x 1 Convolution is a convolution with some special properties in that it can be used for dimensionality reduction,…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Ristretto: Hardware-Oriented Approximation of Convolutional Neural Networks

2016-05-20 · Philipp Gysel

Convolutional neural networks (CNN) have achieved major breakthroughs in recent years. Their performance in computer vision have matched and in some areas even surpassed human capabilities. Deep neural networks can captu…

GPU

Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going

2019-01-21 · Erwei Wang, James J. Davis, Ruizhe Zhao, Ho-Cheung Ng 외

Deep neural networks have proven to be particularly effective in visual and audio recognition tasks. Existing models tend to be computationally expensive and memory intensive, however, and so methods for hardware-oriente…

Hardware-Efficient Softmax and Layer Normalization with Guaranteed Normalization for Edge Devices

2026-04-26 · Dawon Choi, Hana Kim, Ji-Hoon Kim arxiv

In Transformer models, non-GEMM (non-General Matrix Multiplication) operations -- especially Softmax and Layer Normalization (LayerNorm) -- often dominate hardware cost due to their nonlinear nature. To address this, pre…

Phoenix: A Low-Precision Floating-Point Quantization Oriented Architecture for Convolutional Neural Networks

2020-02-29

Convolutional neural networks (CNNs) achieve state-of-the-art performance at the cost of becoming deeper and larger. Although quantization (both fixed-point and floating-point) has proven effective for reducing storage a…

Quantization

Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformers

2024-07-25 · Zhengang Li, Alec Lu, Yanyue Xie, Zhenglun Kong 외

Vision transformers (ViTs) have demonstrated their superior accuracy for computer vision tasks compared to convolutional neural networks (CNNs). However, ViT models are often computation-intensive for efficient deploymen…