TokenMixup: Efficient Attention-guided Token-level Data Augmentation for Transformers

2022-10-14 · Hyeong Kyu Choi, Joonmyung Choi, Hyunwoo J. Kim

Mixup is a commonly adopted data augmentation technique for image classification. Recent advances in mixup methods primarily focus on mixing based on saliency. However, many saliency detectors require intense computation and are especially burdensome for parameter-heavy transformer models. To this end, we propose TokenMixup, an efficient attention-guided token-level data augmentation method that aims to maximize the saliency of a mixed set of tokens. TokenMixup provides x15 faster saliency-aware data augmentation compared to gradient-based methods. Moreover, we introduce a variant of TokenMixup which mixes tokens within a single instance, thereby enabling multi-scale feature augmentation. Experiments show that our methods significantly improve the baseline models' performance on CIFAR and ImageNet-1K, while being more efficient than previous methods. We also reach state-of-the-art performance on CIFAR-100 among from-scratch transformer models. Code is available at https://github.com/mlvlab/TokenMixup.

📄 PDF Abstract BibTeX arXiv:2210.07562

Code (1)

mlvlab/tokenmixup 공식 구현 pytorch

Tasks

Data Augmentation image-classification Image Classification

Methods 이 논문이 사용한 방법론

Mixup Mixup is a data augmentation technique that generates a weighted combination of random image pairs from the training data. Given two images and their ground truth labels:…

Zero-shot sketch-based remote sensing image retrieval based on multi-level and attention-guided tokenization

2024-02-03 · Bo Yang, Chen Wang, Xiaoshuang Ma, Beiping Song 외

Effectively and efficiently retrieving images from remote sensing databases is a critical challenge in the realm of remote sensing big data. Utilizing hand-drawn sketches as retrieval inputs offers intuitive and user-fri…

Cross-Modal Retrieval Image Retrieval Retrieval Zero-Shot Learning

STAR: Stage-Wise Attention-Guided Token Reduction for Efficient Large Vision-Language Models Inference

2025-05-18 · Yichen Guo, Hanze Li, Zonghao Zhang, Jinhao You 외

Although large vision-language models (LVLMs) leverage rich visual token representations to achieve strong performance on multimodal tasks, these tokens also introduce significant computational overhead during inference.…

Token Reduction

Heterogeneous Dependency Graph-Guided Attentionfor Patent Representation Learning

2026-05-11 · Yongmin Yoo, Qiongkai Xu, Zhangkai Wu, Longbing Cao arxiv

Pre-trained language models advance patent classification and retrieval by encoding claims as flat token sequences, but they overlook the dependency hierarchy among claims. Incorporating this hierarchy into self-attentio…

Representation Learning

ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs

2026-06-30 · Yuhao Wang, Mu Qiao, Haiwen Diao, Yunzhi Zhuge 외 arxiv

Multimodal Large Language Models (MLLMs) incur prohibitive inference costs due to long visual token sequences. Training-free visual token reduction provides an efficient solution. However, existing methods distort attent…

Cluster-Level Attention-Guided Parallel Decoding for Masked Diffusion Language Models

2026-05-28 · Heqiang Qi, Wei Huang, Mingyuan Bai, Xiangming Meng arxiv

Masked diffusion language models (MDLMs) enable parallel decoding by predicting all masked positions at each denoising step, yet existing training-free samplers usually decide which positions to commit at token-level gra…