ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks. Current approaches to VLP heavily rely on image feature extraction processes, most of which involve region supervision (e.g., object detection) and the convolutional architecture (e.g., ResNet). Although disregarded in the literature, we find it problematic in terms of both (1) efficiency/speed, that simply extracting input features requires much more computation than the multimodal interaction steps; and (2) expressive power, as it is upper bounded to the expressive power of the visual embedder and its predefined visual vocabulary. In this paper, we present a minimal VLP model, Vision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically simplified to just the same convolution-free manner that we process textual inputs. We show that ViLT is up to tens of times faster than previous VLP models, yet with competitive or better downstream task performance. Our code and pre-trained weights are available at https://github.com/dandelin/vilt.
Code (6)
Tasks
Cross-Modal RetrievalImage RetrievalMultimodal Intent Recognitionmultimodal interactionobject-detectionVisual Question Answering (VQA)Visual ReasoningZero-Shot Cross-Modal RetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
VAuLT: Augmenting the Vision-and-Language Transformer for Sentiment Classification on Social Media
We propose the Vision-and-Augmented-Language Transformer (VAuLT). VAuLT is an extension of the popular Vision-and-Language Transformer (ViLT), and improves performance on vision-and-language (VL) tasks that involve more …
DescriptiveDiversityImage CaptioningLanguage Modelling+3Training Vision-Language Transformers from Captions
Vision-Language Transformers can be learned without low-level human labels (e.g. class labels, bounding boxes, etc). Existing work, whether explicitly utilizing bounding boxes or patches, assumes that the visual backbone…
ViLTA: Enhancing Vision-Language Pre-training through Textual Augmentation
Vision-language pre-training (VLP) methods are blossoming recently, and its crucial goal is to jointly learn visual and textual features via a transformer-based architecture, demonstrating promising improvements on a var…
Image-text matchingLanguage ModelingLanguage ModellingMasked Language Modeling+1ColorFoil: Investigating Color Blindness in Large Vision and Language Models
With the utilization of Transformer architecture, large Vision and Language (V&L) models have shown promising performance in even zero-shot settings. Several studies, however, indicate a lack of robustness of the models …
A Generative Framework for Bidirectional Image-Report Understanding in Chest Radiography
The rapid advancements in large language models (LLMs) have unlocked their potential for multimodal tasks, where text and visual data are processed jointly. However, applying LLMs to medical imaging, particularly for che…
DiagnosticMultimodal Reasoning