Large Batch Training of Convolutional Networks with Layer-wise Adaptive Rate Scaling

2018-01-01 · ICLR 2018 1 · Boris Ginsburg, Igor Gitman, Yang You

A common way to speed up training of large convolutional networks is to add computational units. Training is then performed using data-parallel synchronous Stochastic Gradient Descent (SGD) with a mini-batch divided between computational units. With an increase in the number of nodes, the batch size grows. However, training with a large batch often results in lower model accuracy. We argue that the current recipe for large batch training (linear learning rate scaling with warm-up) is not general enough and training may diverge. To overcome these optimization difficulties, we propose a new training algorithm based on Layer-wise Adaptive Rate Scaling (LARS). Using LARS, we scaled AlexNet and ResNet-50 to a batch size of 16K.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

16k

Methods 이 논문이 사용한 방법론

SPEED The monocular depth estimation (MDE) is the task of estimating depth from a single frame. This information is an essential knowledge in many computer vision tasks such as scene…

1x1 Convolution A 1 x 1 Convolution is a convolution with some special properties in that it can be used for dimensionality reduction,…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Local Response Normalization Local Response Normalization is a normalization layer that implements the idea of lateral inhibition. Lateral inhibition is a concept in neurobiology that refers to the…

Grouped Convolution A Grouped Convolution uses a group of convolutions - multiple kernels per layer - resulting in multiple channel outputs per layer. This leads to wider networks helping a…

ReLU How Do I Communicate to Expedia? How Do I Communicate to Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Live Support & Special Travel…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

LARS Layer-wise Adaptive Rate Scaling, or LARS, is a large batch optimization technique. There are two notable differences between LARS and other adaptive algorithms such as…

Large Batch Training of Convolutional Networks

2017-08-13 · Yang You, Igor Gitman, Boris Ginsburg

The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networks

2023-06-20 · Yuan Cao, Difan Zou, Yuanzhi Li, Quanquan Gu

We study the implicit bias of batch normalization trained by gradient descent. We show that when learning a linear model with batch normalization for binary classification, gradient descent converges to a uniform margin …

Binary Classification

Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks

2019-11-17 · NeurIPS 2019 12 · Difan Zou, Ziniu Hu, Yewen Wang, Song Jiang 외

Graph convolutional networks (GCNs) have recently received wide attentions, due to their successful applications in different graph tasks and different domains. Training GCNs for a large graph, however, is still a challe…

Node Classification

Adaptive Signal Resuscitation: Channel-wise Post-Pruning Repair for Sparse Vision Networks

2026-05-20 · Qishi Zhan, Ziheng Chen, Minxuan Hu arxiv

One-shot magnitude pruning can cause severe accuracy collapse in the high-sparsity regime, even when the pruning mask preserves the largest weights. We argue that this failure reflects a granularity mismatch in post-prun…

Provably Convergent Subgraph-wise Sampling for Fast GNN Training

2023-03-17 · Jie Wang, Zhihao Shi, Xize Liang, Defu Lian 외

Subgraph-wise sampling -- a promising class of mini-batch training techniques for graph neural networks (GNNs -- is critical for real-world applications. During the message passing (MP) in GNNs, subgraph-wise sampling me…