APNN-TC: Accelerating Arbitrary Precision Neural Networks on Ampere GPU Tensor Cores

2021-06-23 · Boyuan Feng, yuke wang, Tong Geng, Ang Li, Yufei Ding

Over the years, accelerating neural networks with quantization has been widely studied. Unfortunately, prior efforts with diverse precisions (e.g., 1-bit weights and 2-bit activations) are usually restricted by limited precision support on GPUs (e.g., int1 and int4). To break such restrictions, we introduce the first Arbitrary Precision Neural Network framework (APNN-TC) to fully exploit quantization benefits on Ampere GPU Tensor Cores. Specifically, APNN-TC first incorporates a novel emulation algorithm to support arbitrary short bit-width computation with int1 compute primitives and XOR/AND Boolean operations. Second, APNN-TC integrates arbitrary precision layer designs to efficiently map our emulation algorithm to Tensor Cores with novel batching strategies and specialized memory organization. Third, APNN-TC embodies a novel arbitrary precision NN design to minimize memory access across layers and further improve performance. Extensive evaluations show that APNN-TC can achieve significant speedup over CUTLASS kernels and various NN models, such as ResNet and VGG.

📄 PDF Abstract BibTeX arXiv:2106.12169

Code (1)

BoyuanFeng/APNN-TC 공식 구현

Tasks

GPUQuantization

Methods 이 논문이 사용한 방법론

ReLU How Do I Communicate to Expedia? How Do I Communicate to Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Live Support & Special Travel…

Batch Normalization 설명 없음

Residual Connection 설명 없음

Average Pooling 설명 없음

Global Average Pooling Global Average Pooling is a pooling operation designed to replace fully connected layers in classical CNNs. The idea is to generate one feature map for each corresponding…

1x1 Convolution A 1 x 1 Convolution is a convolution with some special properties in that it can be used for dimensionality reduction,…

Bottleneck Residual Block A Bottleneck Residual Block is a variant of the residual block that utilises 1x1 convolutions to create a bottleneck. The…

Kaiming Initialization 설명 없음

Accelerating Sparse Deep Neural Networks

2021-04-16 · Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic 외

As neural network model sizes have dramatically increased, so has the interest in various techniques to reduce their parameter counts and accelerate their execution. An active area of research in this field is sparsity -…

GPU Math

Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores

2024-09-26 · Shaobo Ma, Chao Fang, Haikuo Shao, Zhongfeng Wang

Large language models (LLMs) have been widely applied but face challenges in efficient inference. While quantization methods reduce computational demands, ultra-low bit quantization with arbitrary precision is hindered b…

GPU Management Quantization

Toward Capturing Genetic Epistasis From Multivariate Genome-Wide Association Studies Using Mixed-Precision Kernel Ridge Regression

2024-09-03 · Hatem Ltaief, Rabab Alomairy, Qinglei Cao, Jie Ren 외

We exploit the widening margin in tensor-core performance between [FP64/FP32/FP16/INT8,FP64/FP32/FP16/FP8/INT8] on NVIDIA [Ampere,Hopper] GPUs to boost the performance of output accuracy-preserving mixed-precision comput…

CPU GPU

A model-data asymptotic-preserving neural network method based on micro-macro decomposition for gray radiative transfer equations

2022-12-11 · Hongyan Li, Song Jiang, Wenjun Sun, Liwei Xu 외

We propose a model-data asymptotic-preserving neural network(MD-APNN) method to solve the nonlinear gray radiative transfer equations(GRTEs). The system is challenging to be simulated with both the traditional numerical …

APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Acceleration

2025-08-26 · Shaobo Ma, Chao Fang, Haikuo Shao, Zhongfeng Wang arxiv

Large language models (LLMs) have revolutionized AI applications, yet their enormous computational demands severely limit deployment and real-time performance. Quantization methods can help reduce computational costs, ho…