Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
Vision-Language Pre-training (VLP) has advanced the performance of many vision-language tasks, such as image-text retrieval, visual entailment, and visual reasoning. The pre-training mostly utilizes lexical databases and image queries in English. Previous work has demonstrated that the pre-training in English does not transfer well to other languages in a zero-shot setting. However, multilingual pre-trained language models (MPLM) have excelled at a variety of single-modal language tasks. In this paper, we propose a simple yet efficient approach to adapt VLP to unseen languages using MPLM. We utilize a cross-lingual contextualized token embeddings alignment approach to train text encoders for non-English languages. Our approach does not require image input and primarily uses machine translation, eliminating the need for target language data. Our evaluation across three distinct tasks (image-text retrieval, visual entailment, and natural language visual reasoning) demonstrates that this approach outperforms the state-of-the-art multilingual vision-language models without requiring large parallel corpora. Our code is available at https://github.com/Yasminekaroui/CliCoTea.
Code (1)
Tasks
Image-text RetrievalMachine TranslationRetrievalText RetrievalVisual EntailmentVisual ReasoningSimilar Papers 제목 키워드 기반
GemNav: Discrete-Token Visual Robot Navigation using a Multimodal Large Language Model
Visual navigation policies built on large pretrained models have so far followed a common recipe: a dedicated visual encoder, a bespoke action head, and training on thousands of hours of cross-embodiment datasets. We ask…
Visual NavigationRobot NavigationAre You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation
Vision-and-Language Navigation (VLN) requires grounding instructions, such as "turn right and stop at the door", to routes in a visual environment. The actual grounding can connect language to the environment through mul…
Vision and Language NavigationHistopath-C: Towards Realistic Domain Shifts for Histopathology Vision-Language Adaptation
Medical Vision-language models (VLMs) have shown remarkable performances in various medical imaging domains such as histo\-pathology by leveraging pre-trained, contrastive models that exploit visual and textual informati…
Test-time AdaptationEGA: Adapting Frozen Encoders for Vector Search with Bounded Out-of-Distribution Degradation
Vector search systems built on frozen vision encoders face queries from unseen classes at deployment, yet existing adapter training collapses under this shift: high-capacity adapters with global contrastive losses silent…
Dual Adaptive Pyramid Network for Cross-Stain Histopathology Image Segmentation
Supervised semantic segmentation normally assumes the test data being in a similar data domain as the training data. However, in practice, the domain mismatch between the training and unseen data could lead to a signific…
Domain AdaptationImage SegmentationSegmentationSemantic Segmentation