VisLingInstruct: Elevating Zero-Shot Learning in Multi-Modal Language Models with Autonomous Instruction Optimization

2024-02-12 · Dongsheng Zhu, Xunzhu Tang, Weidong Han, Jinghui Lu, Yukun Zhao, Guoliang Xing, Junfeng Wang, Dawei Yin

This paper presents VisLingInstruct, a novel approach to advancing Multi-Modal Language Models (MMLMs) in zero-shot learning. Current MMLMs show impressive zero-shot abilities in multi-modal tasks, but their performance depends heavily on the quality of instructions. VisLingInstruct tackles this by autonomously evaluating and optimizing instructional texts through In-Context Learning, improving the synergy between visual perception and linguistic expression in MMLMs. Alongside this instructional advancement, we have also optimized the visual feature extraction modules in MMLMs, further augmenting their responsiveness to textual content. Our comprehensive experiments on MMLMs, based on FlanT5 and Vicuna, show that VisLingInstruct significantly improves zero-shot performance in visual multi-modal tasks. Notably, it achieves a 13.1% and 9% increase in accuracy over the prior state-of-the-art on the TextVQA and HatefulMemes datasets. Our main code is available at https://github.com/Zhudongsheng75/VisLingInstruct.

📄 PDF Abstract BibTeX arXiv:2402.07398

Code (1)

zhudongsheng75/vislinginstruct 공식 구현 pytorch

Tasks

In-Context LearningTextVQAZero-Shot Learning

Elevating All Zero-Shot Sketch-Based Image Retrieval Through Multimodal Prompt Learning

2024-07-05 · Mainak Singha, Ankit Jha, Divyam Gupta, Pranav Singla 외

We address the challenges inherent in sketch-based image retrieval (SBIR) across various settings, including zero-shot SBIR, generalized zero-shot SBIR, and fine-grained zero-shot SBIR, by leveraging the vision-language …

All Image Retrieval Prompt Learning Retrieval+2

Improving Pediatric Emergency Department Triage with Modality Dropout in Late Fusion Multimodal EHR Models

2026-04-10 · Tyler Yang, Romal Mitr arxiv

Emergency department triage relies heavily on both quantitative vital signs and qualitative clinical notes, yet multimodal machine learning models predicting triage acuity often suffer from modality collapse by over-rely…

Zero-shot Generalization

T-Modules: Translation Modules for Zero-Shot Cross-Modal Machine Translation

2022-05-24 · Paul-Ambroise Duquenne, Hongyu Gong, Benoît Sagot, Holger Schwenk

We present a new approach to perform zero-shot cross-modal transfer between speech and text for translation tasks. Multilingual speech and text are encoded in a joint fixed-size representation space. Then, we compare dif…

Decoder Machine Translation text-to-speech Text to Speech+2

AVGZSLNet: Audio-Visual Generalized Zero-Shot Learning by Reconstructing Label Features from Multi-Modal Embeddings

2020-05-27 · Pratik Mazumder, Pravendra Singh, Kranti Kumar Parida, Vinay P. Namboodiri

In this paper, we propose a novel approach for generalized zero-shot learning in a multi-modal setting, where we have novel classes of audio/video during testing that are not seen during training. We use the semantic rel…

Decoder Generalized Zero-Shot Learning GZSL Video Classification Retrieval+3

DynaIP: Dynamic Image Prompt Adapter for Scalable Zero-shot Personalized Text-to-Image Generation

2025-12-10 · Zhizhong Wang, Tianyi Chu, Zeyi Huang, Nanyang Wang 외 arxiv

Personalized Text-to-Image (PT2I) generation aims to produce customized images based on reference images. A prominent interest pertains to the integration of an image prompt adapter to facilitate zero-shot PT2I without t…

Text-to-Image Generation