Altering Backward Pass Gradients improves Convergence

2021-11-24 · Bishshoy Das, Milton Mondal, Brejesh lall, Shiv Dutt Joshi, Sumantra Dutta Roy

In standard neural network training, the gradients in the backward pass are determined by the forward pass. As a result, the two stages are coupled. This is how most neural networks are trained currently. However, gradient modification in the backward pass has seldom been studied in the literature. In this paper we explore decoupled training, where we alter the gradients in the backward pass. We propose a simple yet powerful method called PowerGrad Transform, that alters the gradients before the weight update in the backward pass and significantly enhances the predictive performance of the neural network. PowerGrad Transform trains the network to arrive at a better optima at convergence. It is computationally extremely efficient, virtually adding no additional cost to either memory or compute, but results in improved final accuracies on both the training and test sets. PowerGrad Transform is easy to integrate into existing training routines, requiring just a few lines of code. PowerGrad Transform accelerates training and makes it possible for the network to better fit the training data. With decoupled training, PowerGrad Transform improves baseline accuracies for ResNet-50 by 0.73%, for SE-ResNet-50 by 0.66% and by more than 1.0% for the non-normalized ResNet-18 network on the ImageNet classification task.

📄 PDF Abstract BibTeX arXiv:2111.12495

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Gradient Clipping One difficulty that arises with optimization of deep neural networks is that large parameter gradients can lead an SGD optimizer to update…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

LMC: Fast Training of GNNs via Subgraph Sampling with Provable Convergence

2023-02-02 · Zhihao Shi, Xize Liang, Jie Wang

The message passing-based graph neural networks (GNNs) have achieved great success in many real-world applications. However, training GNNs on large-scale graphs suffers from the well-known neighbor explosion problem, i.e…

Provably Convergent Subgraph-wise Sampling for Fast GNN Training

2023-03-17 · Jie Wang, Zhihao Shi, Xize Liang, Defu Lian 외

Subgraph-wise sampling -- a promising class of mini-batch training techniques for graph neural networks (GNNs -- is critical for real-world applications. During the message passing (MP) in GNNs, subgraph-wise sampling me…

Spike-based causal inference for weight alignment

2019-10-03 · ICLR 2020 1 · Jordan Guerguiev, Konrad P. Kording, Blake A. Richards

In artificial neural networks trained with gradient descent, the weights used for processing stimuli are also used during backward passes to calculate gradients. For the real brain to approximate gradients, gradient info…

Causal Inference Econometrics

On the Convergence of Stochastic Gradient Descent with Perturbed Forward-Backward Passes

2026-02-24 · Boao Kong, Hengrui Zhang, Kun Yuan arxiv

We study stochastic gradient descent (SGD) for composite optimization problems with $N$ sequential operators subject to perturbations in both the forward and backward passes. Unlike classical analyses that treat gradient…

DropIT: Dropping Intermediate Tensors for Memory-Efficient DNN Training

2022-02-28 · Joya Chen, Kai Xu, Yuhui Wang, Yifei Cheng 외

A standard hardware bottleneck when training deep neural networks is GPU memory. The bulk of memory is occupied by caching intermediate tensors for gradient computation in the backward pass. We propose a novel method to …

GPU Instance Segmentation object-detection Object Detection+1