MVFormer: Diversifying Feature Normalization and Token Mixing for Efficient Vision Transformers
Active research is currently underway to enhance the efficiency of vision transformers (ViTs). Most studies have focused solely on effective token mixers, overlooking the potential relationship with normalization. To boost diverse feature learning, we propose two components: a normalization module called multi-view normalization (MVN) and a token mixer called multi-view token mixer (MVTM). The MVN integrates three differently normalized features via batch, layer, and instance normalization using a learnable weighted sum. Each normalization method outputs a different distribution, generating distinct features. Thus, the MVN is expected to offer diverse pattern information to the token mixer, resulting in beneficial synergy. The MVTM is a convolution-based multiscale token mixer with local, intermediate, and global filters, and it incorporates stage specificity by configuring various receptive fields for the token mixer at each stage, efficiently capturing ranges of visual patterns. We propose a novel ViT model, multi-vision transformer (MVFormer), adopting the MVN and MVTM in the MetaFormer block, the generalized ViT scheme. Our MVFormer outperforms state-of-the-art convolution-based ViTs on image classification, object detection, and instance and semantic segmentation with the same or lower parameters and MACs. Particularly, MVFormer variants, MVFormer-T, S, and B achieve 83.4%, 84.3%, and 84.6% top-1 accuracy, respectively, on ImageNet-1K benchmark.
Code (0)
등록된 구현이 없습니다.
Tasks
image-classificationImage Classificationobject-detectionObject DetectionSemantic SegmentationSpecificityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
I2MVFormer: Large Language Model Generated Multi-View Document Supervision for Zero-Shot Image Classification
Recent works have shown that unstructured text (documents) from online sources can serve as useful auxiliary information for zero-shot image classification. However, these methods require access to a high-quality source …
Classificationimage-classificationImage ClassificationLanguage Modeling+3Decoding Matters: Addressing Amplification Bias and Homogeneity Issue for LLM-based Recommendation
Adapting Large Language Models (LLMs) for recommendation requires careful consideration of the decoding process, given the inherent differences between generating items and natural language. Existing approaches often dir…
DiversityDiversifying Joint Vision-Language Tokenization Learning
Building joint representations across images and text is an essential step for tasks such as Visual Question Answering and Video Question Answering. In this work, we find that the representations must not only jointly ca…
Question AnsweringRepresentation LearningVideo Question AnsweringVisual Question AnsweringTFS-ViT: Token-Level Feature Stylization for Domain Generalization
Standard deep learning models such as convolutional neural networks (CNNs) lack the ability of generalizing to domains which have not been seen during training. This problem is mainly due to the common but often wrong as…
Domain GeneralizationDiversifying the High-level Features for better Adversarial Transferability
Given the great threat of adversarial attacks against Deep Neural Networks (DNNs), numerous works have been proposed to boost transferability to attack real-world applications. However, existing attacks often utilize adv…
Vocal Bursts Intensity Prediction