Sparse GPU Kernels for Deep Learning

2020-06-18 · Trevor Gale, Matei Zaharia, Cliff Young, Erich Elsen

Scientific workloads have traditionally exploited high levels of sparsity to accelerate computation and reduce memory requirements. While deep neural networks can be made sparse, achieving practical speedups on GPUs is difficult because these applications have relatively moderate levels of sparsity that are not sufficient for existing sparse kernels to outperform their dense counterparts. In this work, we study sparse matrices from deep learning applications and identify favorable properties that can be exploited to accelerate computation. Based on these insights, we develop high-performance GPU kernels for two sparse matrix operations widely applicable in neural networks: sparse matrix-dense matrix multiplication and sampled dense-dense matrix multiplication. Our kernels reach 27% of single-precision peak on Nvidia V100 GPUs. Using our kernels, we demonstrate sparse Transformer and MobileNet models that achieve 1.2-2.1x speedups and up to 12.8x memory savings without sacrificing accuracy.

📄 PDF Abstract BibTeX arXiv:2006.10901

Code (1)

google-research/sputnik 공식 구현

Tasks

Deep LearningGPU

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Average Pooling 설명 없음

Pointwise Convolution Pointwise Convolution is a type of convolution that uses a 1x1 kernel: a kernel that iterates through every single point. This…

Depthwise Convolution Depthwise Convolution is a type of convolution where we apply a single convolutional filter for each input channel. In the regular 2D…

Batch Normalization 설명 없음

Global Average Pooling Global Average Pooling is a pooling operation designed to replace fully connected layers in classical CNNs. The idea is to generate one feature map for each corresponding…

Interleaved Structured Sparse Convolutional Neural Networks

2018-06-01 · CVPR 2018 6 · Guotian Xie, Jingdong Wang, Ting Zhang, Jian-Huang Lai 외

In this paper, we study the problem of designing efficient convolutional neural network architectures with the interest in eliminating the redundancy in convolution kernels. In addition to structured sparse kernels, low-…

IGCV$2$: Interleaved Structured Sparse Convolutional Neural Networks

2018-04-17 · Guotian Xie, Jingdong Wang, Ting Zhang, Jian-Huang Lai 외

CoRE Kernels

2014-04-24 · Ping Li

The term "CoRE kernel" stands for correlation-resemblance kernel. In many applications (e.g., vision), the data are often high-dimensional, sparse, and non-binary. We propose two types of (nonlinear) CoRE kernels for non…

General Classification

GPU Kernels for Block-Sparse Weights

2017-12-01 · OpenAi 2017 12 · Scott Gray, Alec Radford and Diederik P. Kingma

We’re releasing highly optimized GPU kernels for an underexplored class of neural network architectures: networks with block-sparse weights. The kernels allow for efficient evaluation and differentiation of linear layers…

GPU Sentiment Analysis

Replicating Kernels with a Short Stride Allows Sparse Reconstructions with Fewer Independent Kernels

2014-06-17 · Peter F. Schultz, Dylan M. Paiton, Wei Lu, Garrett T. Kenyon

In sparse coding it is common to tile an image into nonoverlapping patches, and then use a dictionary to create a sparse representation of each tile independently. In this situation, the overcompleteness of the dictionar…