Dynamic Grained Encoder for Vision Transformers
Transformers, the de-facto standard for language modeling, have been recently applied for vision tasks. This paper introduces sparse queries for vision transformers to exploit the intrinsic spatial redundancy of natural images and save computational costs. Specifically, we propose a Dynamic Grained Encoder for vision transformers, which can adaptively assign a suitable number of queries to each spatial region. Thus it achieves a fine-grained representation in discriminative regions while keeping high efficiency. Besides, the dynamic grained encoder is compatible with most vision transformer frameworks. Without bells and whistles, our encoder allows the state-of-the-art vision transformers to reduce computational complexity by 40%-60% while maintaining comparable performance on image classification. Extensive experiments on object detection and segmentation further demonstrate the generalizability of our approach. Code is available at https://github.com/StevenGrove/vtpack.
Code (1)
Tasks
image-classificationImage ClassificationLanguage ModelingLanguage Modellingobject-detectionObject DetectionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MMFuser: Multimodal Multi-Layer Feature Fuser for Fine-Grained Vision-Language Understanding
Despite significant advancements in Multimodal Large Language Models (MLLMs) for understanding complex human intentions through cross-modal interactions, capturing intricate image details remains challenging. Previous me…
Visual Question AnsweringFIAS: Feature Imbalance-Aware Medical Image Segmentation with Dynamic Fusion and Mixing Attention
With the growing application of transformer in computer vision, hybrid architecture that combine convolutional neural networks (CNNs) and transformers demonstrates competitive ability in medical image segmentation. Howev…
DecoderImage SegmentationMedical Image SegmentationSegmentation+1Contrastive Learning with Bidirectional Transformers for Sequential Recommendation
Contrastive learning with Transformer-based sequence encoder has gained predominance for sequential recommendation. It maximizes the agreements between paired sequence augmentations that share similar semantics. However,…
Contrastive LearningSequential RecommendationDynamic Granularity Matters: Rethinking Vision Transformers Beyond Fixed Patch Splitting
Vision Transformers (ViTs) have demonstrated strong capabilities in capturing global dependencies but often struggle to efficiently represent fine-grained local details. Existing multi-scale approaches alleviate this iss…
Computational EfficiencyGlobal-Local Similarity for Efficient Fine-Grained Image Recognition with Vision Transformers
Fine-grained recognition involves the classification of images from subordinate macro-categories, and it is challenging due to small inter-class differences. To overcome this, most methods perform discriminative feature …
feature selectionFine-Grained Image Recognition