Accelerating Deep Neural Networks via Semi-Structured Activation Sparsity

2023-09-12 · Matteo Grimaldi, Darshan C. Ganji, Ivan Lazarevich, Sudhakar Sah

The demand for efficient processing of deep neural networks (DNNs) on embedded devices is a significant challenge limiting their deployment. Exploiting sparsity in the network's feature maps is one of the ways to reduce its inference latency. It is known that unstructured sparsity results in lower accuracy degradation with respect to structured sparsity but the former needs extensive inference engine changes to get latency benefits. To tackle this challenge, we propose a solution to induce semi-structured activation sparsity exploitable through minor runtime modifications. To attain high speedup levels at inference time, we design a sparse training procedure with awareness of the final position of the activations while computing the General Matrix Multiplication (GEMM). We extensively evaluate the proposed solution across various models for image classification and object detection tasks. Remarkably, our approach yields a speed improvement of $1.25 \times$ with a minimal accuracy drop of $1.1\%$ for the ResNet18 model on the ImageNet dataset. Furthermore, when combined with a state-of-the-art structured pruning method, the resulting models provide a good latency-accuracy trade-off, outperforming models that solely employ structured pruning techniques.

📄 PDF Abstract BibTeX arXiv:2309.06626

Code (0)

등록된 구현이 없습니다.

Tasks

image-classification Image Classification object-detection Object Detection

Methods 이 논문이 사용한 방법론

SPEED The monocular depth estimation (MDE) is the task of estimating depth from a single frame. This information is an essential knowledge in many computer vision tasks such as scene…

Pruning 설명 없음

RT-Lynx: Putting GEMM Sparsity in the Right Place for Diffusion Models

2026-05-26 · Xing Cong, Hanlin Tang, Kan Liu, Lan Tao 외 arxiv

Diffusion Transformers (DiT) achieve strong performance in image generation but incur substantial inference costs. While prior work has reduced this cost via quantization and distillation, semi-structured sparsity, which…

Image Generation

Hardware/Software Co-Design of RISC-V Extensions for Accelerating Sparse DNNs on FPGAs

2025-04-28 · Muhammad Sabih, Abrarul Karim, Jakob Wittmann, Frank Hannig 외

The customizability of RISC-V makes it an attractive choice for accelerating deep neural networks (DNNs). It can be achieved through instruction set extensions and corresponding custom functional units. Yet, efficiently …

Human Detection image-classification Image Classification Keyword Spotting

Reservoir of Importance: Learning Semi-Structured Sparsity with Differentiable Subset Sampling

2026-08-24 · Ha Dinh, Xuan Duy Ta, Khoat Than, Khac-Hoai Nam Bui arxiv

Semi-structured $N$:$M$ sparsity has emerged as a practical direction for accelerating large language models (LLMs). However, existing learnable-mask approaches incur substantial parameter and memory overhead, limiting t…

S2TA: Exploiting Structured Sparsity for Energy-Efficient Mobile CNN Acceleration

2021-07-16 · Zhi-Gang Liu, Paul N. Whatmough, Yuhao Zhu, Matthew Mattina

Exploiting sparsity is a key technique in accelerating quantized convolutional neural network (CNN) inference on mobile devices. Prior sparse CNN accelerators largely exploit un-structured sparsity and achieve significan…

Motivating Next-Gen Accelerators with Flexible (N:M) Activation Sparsity via Benchmarking Lightweight Post-Training Sparsification Approaches

2025-09-26 · Shirin Alanova, Kristina Kazistova, Ekaterina Galaeva, Alina Kostromina 외 arxiv

The demand for efficient large language model (LLM) inference has intensified the focus on sparsification techniques. While semi-structured (N:M) pruning is well-established for weights, its application to activation pru…