Papers Zero-Shot Cross-Modal Retrieval
“Zero-Shot Cross-Modal Retrieval” 태그가 달린 논문 28편 · 필터 해제
MatBind: A Shared Embedding Space for Multimodal Materials Characterization
Fully characterizing a crystalline material requires integrating heterogeneous data sources -- atomic structures, diffraction patterns, electronic density of states, and natural language -- each of which captures a diffe…
Zero-Shot Cross-Modal RetrievalContrastive LearningSTAR: Semantic-Traffic Alignment and Retrieval for Zero-Shot HTTPS Website Fingerprinting
Modern HTTPS mechanisms such as Encrypted Client Hello (ECH) and encrypted DNS improve privacy but remain vulnerable to website fingerprinting (WF) attacks, where adversaries infer visited sites from encrypted traffic pa…
Zero-Shot Cross-Modal RetrievalFineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs
As a pioneering vision-language model, CLIP (Contrastive Language-Image Pre-training) has achieved significant success across various domains and a wide range of downstream vision-language tasks. However, the text encode…
cross-modal alignmentCross-Modal RetrievalImage GenerationText to Image Generation+2A Recipe for Improving Remote Sensing VLM Zero Shot Generalization
Foundation models have had a significant impact across various AI applications, enabling use cases that were previously impossible. Contrastive Visual Language Models (VLMs), in particular, have outperformed other techni…
Cross-Modal RetrievalZero-Shot Cross-Modal RetrievalZero-shot GeneralizationIMPACT: A Large-scale Integrated Multimodal Patent Analysis and Creation Dataset for Design Patents
In this paper, we introduce IMPACT (Integrated Multimodal Patent Analysis and Creation Dataset for Design Patents), a large-scale multimodal patent dataset with detailed captions for design patent figures. Our dataset in…
Cross-Modal RetrievalImage ClassificationImage RetrievalPatent classification+5COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training
Vision-Language Models (VLMs) trained with contrastive loss have achieved significant advancements in various vision and language tasks. However, the global nature of the contrastive loss makes VLMs focus predominantly o…
Self-Supervised LearningSemantic SegmentationUnsupervised Semantic Segmentation with Language-image Pre-trainingZero-Shot Cross-Modal Retrieval+1Merlin: A Vision Language Foundation Model for 3D Computed Tomography
Over 85 million computed tomography (CT) scans are performed annually in the US, of which approximately one quarter focus on the abdomen. Given the current radiologist shortage, there is a large impetus to use artificial…
3D Semantic SegmentationComputed Tomography (CT)Cross-Modal RetrievalDisease Prediction+4M2-Encoder: Advancing Bilingual Image-Text Understanding by Large-scale Efficient Pretraining
Vision-language foundation models like CLIP have revolutionized the field of artificial intelligence. Nevertheless, VLM models supporting multi-language, e.g., in both Chinese and English, have lagged due to the relative…
GPUzero-shot-classificationZero-Shot Cross-Modal RetrievalZero-shot Image Retrieval+3InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
The exponential growth of large language models (LLMs) has opened up numerous possibilities for multimodal AGI systems. However, the progress in vision and vision-language foundation models, which are also critical eleme…
Image RetrievalImage-to-Text RetrievalLanguage ModellingLarge Language Model+11Implicit Differentiable Outlier Detection Enable Robust Deep Multimodal Analysis
Deep network models are often purely inductive during both training and inference on unseen data. When these models are used for prediction, but they may fail to capture important semantic information and implicit depend…
Cross-Modal RetrievalImage CaptioningImage RetrievalNatural Language Understanding+5VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset
Vision and text have been fully explored in contemporary video-text foundational models, while other modalities such as audio and subtitles in videos have not received sufficient attention. In this paper, we resort to es…
Audio captioningAudio-Visual CaptioningAudio-visual Question AnsweringCross-Modal Retrieval+12Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers
We present Region-aware Open-vocabulary Vision Transformers (RO-ViT) - a contrastive image-text pretraining recipe to bridge the gap between image-level pretraining and open-vocabulary object detection. At the pretrainin…
Contrastive LearningImage-text Retrievalobject-detectionObject Detection+5Position-guided Text Prompt for Vision-Language Pre-training
Vision-Language Pre-Training (VLP) has shown promising capabilities to align image and text pairs, facilitating a broad variety of cross-modal learning tasks. However, we observe that VLP models often lack the visual gro…
Cross-Modal RetrievalImage CaptioningObjectPosition+4Reproducible scaling laws for contrastive language-image learning
Scaling up neural networks has led to remarkable performance across a wide range of tasks. Moreover, performance often follows reliable scaling laws as a function of training set size, model size, and compute, which offe…
Image ClassificationOpen Vocabulary Attribute DetectionRetrievalzero-shot-classification+3AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities
In this work, we present a conceptually simple and effective method to train a strong bilingual/multilingual multimodal representation model. Starting from the pre-trained multimodal representation model CLIP released by…
Contrastive LearningCross-Modal RetrievalImage ClassificationImage Retrieval+9ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training
Recent Vision-Language Pre-trained (VLP) models based on dual encoder have attracted extensive attention from academia and industry due to their superior performance on various cross-modal tasks and high computational ef…
Computational EfficiencyContrastive LearningCross-Modal RetrievalImage Retrieval+5Information-Theoretic Hashing for Zero-Shot Cross-Modal Retrieval
Zero-shot cross-modal retrieval (ZS-CMR) deals with the retrieval problem among heterogenous data from unseen classes. Typically, to guarantee generalization, the pre-defined class embeddings from natural language proces…
Cross-Modal RetrievalRetrievalZero-Shot Cross-Modal RetrievalImage as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
A big convergence of language, vision, and multimodal pretraining is emerging. In this work, we introduce a general-purpose multimodal foundation model BEiT-3, which achieves state-of-the-art transfer performance on both…
AllCross-Modal RetrievalImage Captioningimage-classification+13CoCa: Contrastive Captioners are Image-Text Foundation Models
Exploring large-scale pretrained foundation models is of significant interest in computer vision because these models can be quickly transferred to many downstream tasks. This paper presents Contrastive Captioner (CoCa),…
Action ClassificationDecoderImage CaptioningImage Classification+9Flamingo: a Visual Language Model for Few-Shot Learning
Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, a family of Visual Language Mode…
Few-Shot LearningGenerative Visual Question AnsweringLanguage ModelingLanguage Modelling+12