DropIT: Dropping Intermediate Tensors for Memory-Efficient DNN Training

2022-02-28 · Joya Chen, Kai Xu, Yuhui Wang, Yifei Cheng, Angela Yao

A standard hardware bottleneck when training deep neural networks is GPU memory. The bulk of memory is occupied by caching intermediate tensors for gradient computation in the backward pass. We propose a novel method to reduce this footprint - Dropping Intermediate Tensors (DropIT). DropIT drops min-k elements of the intermediate tensors and approximates gradients from the sparsified tensors in the backward pass. Theoretically, DropIT reduces noise on estimated gradients and therefore has a higher rate of convergence than vanilla-SGD. Experiments show that we can drop up to 90\% of the intermediate tensor elements in fully-connected and convolutional layers while achieving higher testing accuracy for Visual Transformers and Convolutional Neural Networks on various tasks (e.g., classification, object detection, instance segmentation). Our code and models are available at https://github.com/chenjoya/dropit.

📄 PDF Abstract BibTeX arXiv:2202.13808

Code (1)

ChenJoya/dropit 공식 구현 pytorch

Tasks

GPUInstance Segmentation object-detection Object Detection Semantic Segmentation

Methods 이 논문이 사용한 방법론

Attention 설명 없음

ReLU How Do I Communicate to Expedia? How Do I Communicate to Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Live Support & Special Travel…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Average Pooling 설명 없음

Batch Normalization 설명 없음

Global Average Pooling Global Average Pooling is a pooling operation designed to replace fully connected layers in classical CNNs. The idea is to generate one feature map for each corresponding…

Residual Block Residual Blocks are skip-connection blocks that learn residual functions with reference to the layer inputs, instead of learning unreferenced functions. They were introduced…

Max Pooling Max Pooling is a pooling operation that calculates the maximum value for patches of a feature map, and uses it to create a downsampled (pooled) feature map. It is usually…

TACO: Efficient Communication Compression of Intermediate Tensors for Scalable Tensor-Parallel LLM Training

2026-04-27 · Man Liu, Xingchen Liu, Xingjian Tian, Bing Lu 외 arxiv

Handling communication overhead in large-scale tensor-parallel training remains a critical challenge due to the dense, near-zero distributions of intermediate tensors, which exacerbate errors under frequent communication…

Improving compute efficacy frontiers with SliceOut

2020-07-21 · Pascal Notin, Aidan N. Gomez, Joanna Yoo, Yarin Gal

Pushing forward the compute efficacy frontier in deep learning is critical for tasks that require frequent model re-training or workloads that entail training a large number of models. We introduce SliceOut -- a dropout-…

Deep Learning GPU

Improving Continual Learning for Gaussian Splatting based Environments Reconstruction on Commercial Off-the-Shelf Edge Devices

2026-03-09 · Ivan Zaino, Matteo Risso, Daniele Jahier Pagliari, Miguel de Prado 외 arxiv

Novel view synthesis (NVS) is increasingly relevant for edge robotics, where compact and incrementally updatable 3D scene models are needed for SLAM, navigation, and inspection under tight memory and latency budgets. Var…

Novel View Synthesis Continual Learning

MoEBlaze: Breaking the Memory Wall for Efficient MoE Training on Modern GPUs

2026-01-08 · Jiyuan Zhang, Yining Liu, Siqi Yan, Lisen Deng 외 arxiv

The pervasive "memory wall" bottleneck is significantly amplified in modern large-scale Mixture-of-Experts (MoE) architectures. MoE's inherent architectural sparsity leads to sparse arithmetic compute and also introduces…

Survey on Large Scale Neural Network Training

2022-02-21 · Julia Gusak, Daria Cherniuk, Alena Shilova, Alexander Katrutsa 외

Modern Deep Neural Networks (DNNs) require significant memory to store weight, activations, and other intermediate tensors during training. Hence, many models do not fit one GPU device or can be trained using only a smal…

GPU Survey