Jina CLIP: Your CLIP Model Is Also Your Text Retriever
Contrastive Language-Image Pretraining (CLIP) is widely used to train models to align images and texts in a common embedding space by mapping them to fixed-sized vectors. These models are key to multimodal information retrieval and related tasks. However, CLIP models generally underperform in text-only tasks compared to specialized text models. This creates inefficiencies for information retrieval systems that keep separate embeddings and models for text-only and multimodal tasks. We propose a novel, multi-task contrastive training method to address this issue, which we use to train the jina-clip-v1 model to achieve the state-of-the-art performance on both text-image and text-text retrieval tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Information RetrievalRetrievalText RetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
jina-clip-v2: Multilingual Multimodal Embeddings for Text and Images
Contrastive Language-Image Pretraining (CLIP) has been widely used for crossmodal information retrieval and multimodal understanding tasks. However, CLIP models are mainly optimized for crossmodal vision-language tasks a…
Contrastive LearningCross-Modal Information RetrievalInformation RetrievalRepresentation Learning+3Clip Your Sequences Fairly: Enforcing Length Fairness for Sequence-Level RL
We propose FSPO (Fair Sequence Policy Optimization), a sequence-level reinforcement learning method for LLMs that enforces length-fair clipping on the importance-sampling (IS) weight. We study RL methods with sequence-le…
Reinforcement LearningHow (Mis)calibrated is Your Federated CLIP and What To Do About It?
While vision-language models like CLIP have been extensively studied, their calibration, crucial for reliable predictions, has received limited attention. Although a few prior works have examined CLIP calibration in offl…
Federated LearningClipBot: an educational, physically impaired robot that learns to walk via genetic algorithm optimization
Educational robots allow experimenting with a variety of principles from mechanics, electronics, and informatics. Here we propose ClipBot, a low-cost, do-it-yourself, robot whose skeleton is made of two paper clips. An A…
CAPE: A CLIP-Aware Pointing Ensemble of Complementary Heatmap Cues for Embodied Reference Understanding
We address Embodied Reference Understanding, the task of predicting the object a person in the scene refers to through pointing gesture and language. This requires multimodal reasoning over text, visual pointing cues, an…
Multimodal Reasoning