Boosting vision transformers for image retrieval
Vision transformers have achieved remarkable progress in vision tasks such as image classification and detection. However, in instance-level image retrieval, transformers have not yet shown good performance compared to convolutional networks. We propose a number of improvements that make transformers outperform the state of the art for the first time. (1) We show that a hybrid architecture is more effective than plain transformers, by a large margin. (2) We introduce two branches collecting global (classification token) and local (patch tokens) information, from which we form a global image representation. (3) In each branch, we collect multi-layer features from the transformer encoder, corresponding to skip connections across distant layers. (4) We enhance locality of interactions at the deeper layers of the encoder, which is the relative weakness of vision transformers. We train our model on all commonly used training sets and, for the first time, we make fair comparisons separately per training set. In all cases, we outperform previous models based on global representation. Public code is available at https://github.com/dealicious-inc/DToP.
Code (1)
Tasks
image-classificationImage ClassificationImage RetrievalRetrievalSimilar Papers 제목 키워드 기반
Training Vision Transformers for Image Retrieval
Transformers have shown outstanding results for natural language understanding and, more recently, for image classification. We here extend this work and propose a transformer-based approach for image retrieval: we adopt…
image-classificationImage ClassificationImage RetrievalMetric Learning+2KVT: k-NN Attention for Boosting Vision Transformers
Convolutional Neural Networks (CNNs) have dominated computer vision for years, due to its ability in capturing locality and translation invariance. Recently, many vision transformer architectures have been proposed and t…
VLDeformer: Vision-Language Decomposed Transformer for Fast Cross-Modal Retrieval
Cross-model retrieval has emerged as one of the most important upgrades for text-only search engines (SE). Recently, with powerful representation for pairwise text-image inputs via early interaction, the accuracy of visi…
Contrastive LearningCross-Modal RetrievalImage RetrievalRetrievalVisual Representation Learning with Self-Supervised Attention for Low-Label High-data Regime
Self-supervision has shown outstanding results for natural language processing, and more recently, for image recognition. Simultaneously, vision transformers and its variants have emerged as a promising and scalable alte…
Few-Shot Image Classificationimage-classificationImage ClassificationImage Retrieval+4General Image Descriptors for Open World Image Retrieval using ViT CLIP
The Google Universal Image Embedding (GUIE) Challenge is one of the first competitions in multi-domain image representations in the wild, covering a wide distribution of objects: landmarks, artwork, food, etc. This is a …
Image RetrievalRetrievalZero-Shot Image ClassificationZero-shot Image Retrieval+1