Optimizing Block-Sparse Matrix Multiplications on CUDA with TVM
We implemented and optimized matrix multiplications between dense and block-sparse matrices on CUDA. We leveraged TVM, a deep learning compiler, to explore the schedule space of the operation and generate efficient CUDA code. With the automatic parameter tuning in TVM, our cross-thread reduction based implementation achieved competitive or better performance compared with other state-of-the-art frameworks.
Code (1)
Similar Papers 제목 키워드 기반
Secure Sparse Matrix Multiplications and their Applications to Privacy-Preserving Machine Learning
To preserve data privacy, multi-party computation (MPC) enables executing Machine Learning (ML) algorithms on private data. However, MPC frameworks do not include optimized operations on sparse data. This absence makes t…
Block-wise Dynamic Sparseness
Neural networks have achieved state of the art performance across a wide variety of machine learning tasks, often with large and computation-heavy models. Inducing sparseness as a way to reduce the memory and computation…
Language ModelingLanguage ModellingNeural incomplete factorization: learning preconditioners for the conjugate gradient method
The convergence of the conjugate gradient method for solving large-scale and sparse linear equation systems depends on the spectral properties of the system matrix, which can be improved by preconditioning. In this paper…
Computational EfficiencySpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference
An efficient attention implementation is essential for large models due to its quadratic time complexity. Fortunately, attention commonly exhibits sparsity, i.e., many values in the attention map are near zero, allowing …
modelVideo GenerationPopSparse: Accelerated block sparse matrix multiplication on IPU
Reducing the computational cost of running large scale neural networks using sparsity has attracted great attention in the deep learning community. While much success has been achieved in reducing FLOP and parameter coun…
GPU