Pushing the Limits of BFP on Narrow Precision LLM Inference

2025-01-21 · Hui Wang, Yuan Cheng, Xiaomeng Han, Zhengpeng Zhao, Dawei Yang, Zhe Jiang

The substantial computational and memory demands of Large Language Models (LLMs) hinder their deployment. Block Floating Point (BFP) has proven effective in accelerating linear operations, a cornerstone of LLM workloads. However, as sequence lengths grow, nonlinear operations, such as Attention, increasingly become performance bottlenecks due to their quadratic computational complexity. These nonlinear operations are predominantly executed using inefficient floating-point formats, which renders the system challenging to optimize software efficiency and hardware overhead. In this paper, we delve into the limitations and potential of applying BFP to nonlinear operations. Given our findings, we introduce a hardware-software co-design framework (DB-Attn), including: (i) DBFP, an advanced BFP version, overcomes nonlinear operation challenges with a pivot-focus strategy for diverse data and an adaptive grouping strategy for flexible exponent sharing. (ii) DH-LUT, a novel lookup table algorithm dedicated to accelerating nonlinear operations with DBFP format. (iii) An RTL-level DBFP-based engine is implemented to support DB-Attn, applicable to FPGA and ASIC. Results show that DB-Attn provides significant performance improvements with negligible accuracy loss, achieving 74% GPU speedup on Softmax of LLaMA and 10x low overhead performance improvement over SOTA designs.

📄 PDF Abstract BibTeX arXiv:2502.00026

Code (0)

등록된 구현이 없습니다.

Tasks

GPU

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

LLaMA LLaMA is a collection of foundation language models ranging from 7B to 65B parameters. It is based on the transformer architecture with various improvements that were…

SKIM: Any-bit Quantization Pushing The Limits of Post-Training Quantization

2024-12-05 · Runsheng Bai, Bo Liu, Qiang Liu

Large Language Models (LLMs) exhibit impressive performance across various tasks, but deploying them for inference poses challenges. Their high resource demands often necessitate complex, costly multi-GPU pipelines, or t…

Clustering GPU Quantization

CascadeCNN: Pushing the Performance Limits of Quantisation in Convolutional Neural Networks

2018-07-13 · Alexandros Kouris, Stylianos I. Venieris, Christos-Savvas Bouganis

This work presents CascadeCNN, an automated toolflow that pushes the quantisation limits of any given CNN model, aiming to perform high-throughput inference. A two-stage architecture tailored for any given CNN-FPGA pair …

Pushing the Limits of Narrow Precision Inferencing at Cloud Scale with Microsoft Floating Point

2020-12-01 · NeurIPS 2020 12 · Bita Darvish Rouhani, Daniel Lo, Ritchie Zhao, Ming Liu 외

In this paper, we explore the limits of Microsoft Floating Point (MSFP), a new class of datatypes developed for production cloud-scale inferencing on custom hardware. Through the co-evolution of hardware design and algor…

image-classification Image Classification Question Answering

CascadeCNN: Pushing the performance limits of quantisation

2018-05-22 · Alexandros Kouris, Stylianos I. Venieris, Christos-Savvas Bouganis

This work presents CascadeCNN, an automated toolflow that pushes the quantisation limits of any given CNN model, to perform high-throughput inference by exploiting the computation time-accuracy trade-off. Without the nee…

Pushing the Limits of Beam Search Decoding for Transducer-based ASR models

2025-05-30 · Lilit Grigoryan, Vladimir Bataev, Andrei Andrusenko, Hainan Xu 외

Transducer models have emerged as a promising choice for end-to-end ASR systems, offering a balanced trade-off between recognition accuracy, streaming capabilities, and inference speed in greedy decoding. However, beam s…

GPU