Training Noise Token Pruning

2024-11-27 · Mingxing Rao, Bohan Jiang, Daniel Moyer

In the present work we present Training Noise Token (TNT) Pruning for vision transformers. Our method relaxes the discrete token dropping condition to continuous additive noise, providing smooth optimization in training, while retaining discrete dropping computational gains in deployment settings. We provide theoretical connections to Rate-Distortion literature, and empirical evaluations on the ImageNet dataset using ViT and DeiT architectures demonstrating TNT's advantages over previous pruning methods.

📄 PDF Abstract BibTeX arXiv:2411.18092

Code (1)

mx-ethan-rao/tnt 공식 구현 pytorch

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Feedforward Network A Feedforward Network, or a Multilayer Perceptron (MLP), is a neural network with solely densely connected layers. This is the classic neural network architecture of the…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Multi-Head Attention 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gating

2026-03-07 · Landi He, Xiaoyu Yang, Lijian Xu arxiv

Visual tokens dominate inference cost in vision-language models (VLMs), yet many carry redundant information. Existing pruning methods alleviate this but typically rely on attention magnitude or similarity scores. We ref…

Combating Textual Noise and Redundancy: Entropy-Aware Dense Visual Token Pruning

2026-07-02 · Xuehui Wang, Xuankun Yang, Wei Shen arxiv

Visual token pruning is a crucial strategy for accelerating VLMs by compressing redundant image patches, yet existing methods often fail to preserve critical cues under dense instructions and fine-grained queries. In thi…

Beyond Surrogate Gradients: Fully Differentiable Token Pruning for Vision-Language Models

2026-05-27 · Landi He, Mingde Yao, Shawn Young, Lijian Xu arxiv

Visual token pruning reduces the computational cost of Vision-Language Models (VLMs) by removing redundant visual tokens. Existing methods typically rely on Gumbel-Softmax to approximate discrete selection during trainin…

Continuous Control

OptiPrune: Boosting Prompt-Image Consistency with Attention-Guided Noise and Dynamic Token Selection

2025-07-01 · Ziji Lu arxiv

Text-to-image diffusion models often struggle to achieve accurate semantic alignment between generated images and text prompts while maintaining efficiency for deployment on resource-constrained hardware. Existing approa…

CAT Pruning: Cluster-Aware Token Pruning For Text-to-Image Diffusion Models

2025-02-01 · Xinle Cheng, Zhuoming Chen, Zhihao Jia

Diffusion models have revolutionized generative tasks, especially in the domain of text-to-image synthesis; however, their iterative denoising process demands substantial computational resources. In this paper, we presen…

Denoising Image Generation