Improving compute efficacy frontiers with SliceOut

2020-07-21 · Pascal Notin, Aidan N. Gomez, Joanna Yoo, Yarin Gal

Pushing forward the compute efficacy frontier in deep learning is critical for tasks that require frequent model re-training or workloads that entail training a large number of models. We introduce SliceOut -- a dropout-inspired scheme designed to take advantage of GPU memory layout to train deep learning models faster without impacting final test accuracy. By dropping contiguous sets of units at random, our method realises training speedups through (1) fast memory access and matrix multiplication of smaller tensors, and (2) memory savings by avoiding allocating memory to zero units in weight gradients and activations. At test time, turning off SliceOut performs an implicit ensembling across a linear number of architectures that preserves test accuracy. We demonstrate 10-40% speedups and memory reduction with Wide ResNets, EfficientNets, and Transformer models, with minimal to no loss in accuracy. This leads to faster processing of large computational workloads overall, and significantly reduce the resulting energy consumption and CO2emissions.

📄 PDF Abstract BibTeX arXiv:2007.10909

Code (0)

등록된 구현이 없습니다.

Tasks

Deep LearningGPU

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Multi-Head Attention 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Divergence Frontiers for Generative Models: Sample Complexity, Quantization Effects, and Frontier Integrals

2021-06-15 · NeurIPS 2021 12 · Lang Liu, Krishna Pillutla, Sean Welleck, Sewoong Oh 외

The spectacular success of deep generative models calls for quantitative tools to measure their statistical performance. Divergence frontiers have recently been proposed as an evaluation framework for generative models, …

Diversity Quantization

Friction

Improving compute efficacy frontiers with SliceOut

Code (0)

Tasks

Methods 이 논문이 사용한 방법론

Similar Papers 제목 키워드 기반

Divergence Frontiers for Generative Models: Sample Complexity, Quantization Effects, and Frontier Integrals

Frontiers and Exact Learning of ELI Queries under DL-Lite Ontologies

Frontiers in Evolutionary Computation: A Workshop Report

Frontiers in Collective Intelligence: A Workshop Report

Explaining the emergence of land-use frontiers