Accelerator-driven Data Arrangement to Minimize Transformers Run-time on Multi-core Architectures

2023-12-20 · Alireza Amirshahi, Giovanni Ansaloni, David Atienza

The increasing complexity of transformer models in artificial intelligence expands their computational costs, memory usage, and energy consumption. Hardware acceleration tackles the ensuing challenges by designing processors and accelerators tailored for transformer models, supporting their computation hotspots with high efficiency. However, memory bandwidth can hinder improvements in hardware accelerators. Against this backdrop, in this paper we propose a novel memory arrangement strategy, governed by the hardware accelerator's kernel size, which effectively minimizes off-chip data access. This arrangement is particularly beneficial for end-to-end transformer model inference, where most of the computation is based on general matrix multiplication (GEMM) operations. Additionally, we address the overhead of non-GEMM operations in transformer models within the scope of this memory data arrangement. Our study explores the implementation and effectiveness of the proposed accelerator-driven data arrangement approach in both single- and multi-core systems. Our evaluation demonstrates that our approach can achieve up to a 2.8x speed increase when executing inferences employing state-of-the-art transformers.

📄 PDF Abstract BibTeX arXiv:2312.13000

Code (1)

gem5-x/tic-sat 공식 구현

Methods 이 논문이 사용한 방법론

SPEED The monocular depth estimation (MDE) is the task of estimating depth from a single frame. This information is an essential knowledge in many computer vision tasks such as scene…

ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformers

2023-07-07 · Gamze İslamoğlu, Moritz Scherer, Gianna Paulin, Tim Fischer 외

Transformer networks have emerged as the state-of-the-art approach for natural language processing tasks and are gaining popularity in other domains such as computer vision and audio processing. However, the efficient ha…

Quantization

PDA-LSTM: Knowledge-driven page data arrangement based on LSTM for LCM supression in QLC 3D NAND flash memories

2025-10-29 · Qianhui Li, Weiya Wang, Qianqi Zhao, Tong Qu 외 arxiv

Quarter level cell (QLC) 3D NAND flash memory is emerging as the predominant storage solution in the era of artificial intelligence. QLC 3D NAND flash stores 4 bit per cell to expand the storage density, resulting in nar…

Accelerating Post-Quantum Cryptography via LLM-Driven Hardware-Software Co-Design

2026-02-10 · Yuchao Liao, Tosiron Adegbija, Roman Lysecky arxiv

Post-quantum cryptography (PQC) is crucial for securing data against emerging quantum threats. However, its algorithms are computationally complex and difficult to implement efficiently on hardware. In this paper, we exp…

ME-ViT: A Single-Load Memory-Efficient FPGA Accelerator for Vision Transformers

2024-02-15 · Kyle Marino, Pengmiao Zhang, Viktor Prasanna

Vision Transformers (ViTs) have emerged as a state-of-the-art solution for object classification tasks. However, their computational demands and high parameter count make them unsuitable for real-time inference, promptin…

GPU

SwiftTron: An Efficient Hardware Accelerator for Quantized Transformers

2023-04-08 · Alberto Marchisio, Davide Dura, Maurizio Capra, Maurizio Martina 외

Transformers' compute-intensive operations pose enormous challenges for their deployment in resource-constrained EdgeAI / tinyML devices. As an established neural network compression technique, quantization reduces the h…

Neural Network Compression Quantization