Efficient Dynamic Structured Sparse Training with Learned Shuffles

2025-10-16 · Abhishek Tyagi, Arjun Iyer, Liam Young, William H Renninger, Christopher Kanan, Yuhao Zhu arxiv

Structured sparsity accelerates training and inference on modern GPUs, yet it still trails unstructured dynamic sparse training (DST) in accuracy. The shortfall stems from a loss of expressivity: whereas a dense layer can realize every possible mask obtained by choosing any $w$ active weights out of $n$, a fixed block or N:M layout explores only a subset of those possibilities. We propose to close this gap by learning, for each layer, a single permutation matrix jointly with the structured weight matrix. Applied to three canonical structures -- block, N:M, and diagonals -- we show that permutation-augmented DST (PA-DST) matches unstructured baselines (RigL, SET) at 90--95\% sparsity on ImageNet-1K (ViT-B/16) and WikiText-103 (GPT-2), yet trains up to $1.21\times$ and infers up to $2.9\times$ faster. The results position structure + learned permutation as a sweet spot between accuracy and efficiency.

📄 PDF Abstract BibTeX arXiv:2510.14812

Code (0)

등록된 구현이 없습니다.

Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps

2020-12-29 · ICLR 2020 1 · Tri Dao, Nimit S. Sohoni, Albert Gu, Matthew Eichhorn 외

Modern neural network architectures use structured linear transformations, such as low-rank matrices, sparse matrices, permutations, and the Fourier transform, to improve inference speed and reduce memory usage compared …

All image-classification Image Classification speech-recognition+1

Sparse Neural Networks Topologies

2017-06-18 · Alfred Bourely, John Patrick Boueri, Krzysztof Choromonski

We propose Sparse Neural Network architectures that are based on random or structured bipartite graph topologies. Sparse architectures provide compression of the models learned and speed-ups of computations, they can als…

Dynamic Sparse Training with Structured Sparsity

2023-05-03 · Mike Lasby, Anna Golubeva, Utku Evci, Mihai Nica 외

Dynamic Sparse Training (DST) methods achieve state-of-the-art results in sparse neural network training, matching the generalization of dense models while enabling sparse training and inference. Although the resulting m…

CPU GPU

ShuffleSeg: Real-time Semantic Segmentation Network

2018-03-10 · Mostafa Gamal, Mennatullah Siam, Moemen Abdel-Razek

Real-time semantic segmentation is of significant importance for mobile and robotics related applications. We propose a computationally efficient segmentation network which we term as ShuffleSeg. The proposed architectur…

Real-Time Semantic Segmentation Segmentation Semantic Segmentation

Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researchers

2023-02-06 · Shiwei Liu, Zhangyang Wang

This article does not propose any novel algorithm or new hardware for sparsity. Instead, it aims to serve the "common good" for the increasingly prosperous Sparse Neural Network (SNN) research community. We attempt to su…

General Knowledge