Rethinking Sparse Lexical Representations for Image Retrieval in the Age of Rising Multi-Modal Large Language Models

2024-08-29 · Kengo Nakata, Daisuke Miyashita, Youyang Ng, Yasuto Hoshi, Jun Deguchi

In this paper, we rethink sparse lexical representations for image retrieval. By utilizing multi-modal large language models (M-LLMs) that support visual prompting, we can extract image features and convert them into textual data, enabling us to utilize efficient sparse retrieval algorithms employed in natural language processing for image retrieval tasks. To assist the LLM in extracting image features, we apply data augmentation techniques for key expansion and analyze the impact with a metric for relevance between images and textual data. We empirically show the superior precision and recall performance of our image retrieval method compared to conventional vision-language model-based methods on the MS-COCO, PASCAL VOC, and NUS-WIDE datasets in a keyword-based image retrieval scenario, where keywords serve as search queries. We also demonstrate that the retrieval performance can be improved by iteratively incorporating keywords into search queries.

📄 PDF Abstract BibTeX arXiv:2408.16296

Code (0)

등록된 구현이 없습니다.

Tasks

Data Augmentation Image RetrievalLanguage ModelingLanguage Modelling RetrievalVisual Prompting

Sparsifying Sparse Representations for Passage Retrieval by Top-$k$ Masking

2021-12-17 · Jheng-Hong Yang, Xueguang Ma, Jimmy Lin

Sparse lexical representation learning has demonstrated much progress in improving passage retrieval effectiveness in recent models such as DeepImpact, uniCOIL, and SPLADE. This paper describes a straightforward yet effe…

Passage Retrieval Representation Learning Retrieval Self-Learning

Inference-Free Multimodal Learned Sparse Retrieval for Production-Scale Visual Document Search

2026-05-29 · Gyu-Hwung Cho, Youngjune Lee, Kiyoon Jeong, Siyoung Lee 외 arxiv

As large-scale visual-document corpora such as arXiv papers and enterprise PDFs continue to grow, visual-document retrieval has gained increasing attention; yet it still lacks a deployable system that lexically indexes v…

LSTM-based Selective Dense Text Retrieval Guided by Sparse Lexical Retrieval

2025-02-15 · Yingrui Yang, Parker Carlson, Yifan Qiao, Wentai Xie 외

This paper studies fast fusion of dense retrieval and sparse lexical retrieval, and proposes a cluster-based selective dense retrieval method called CluSD guided by sparse lexical retrieval. CluSD takes a lightweight clu…

Retrieval Text Retrieval

Rescaling MLM-Head for Neural Sparse Retrieval

2026-06-17 · Youngjoon Jang, Seongtae Hong, Jonah Turner, Heuiseok Lim arxiv

Learned sparse retrieval (LSR) models such as SPLADE have traditionally used BERT-style masked language models as backbone encoders. A natural expectation is that replacing BERT with stronger pretrained encoders should i…

Dual Skipping Guidance for Document Retrieval with Learned Sparse Representations

2022-04-23 · Yifan Qiao, Yingrui Yang, Haixin Lin, Tianbo Xiong 외

This paper proposes a dual skipping guidance scheme with hybrid scoring to accelerate document retrieval that uses learned sparse representations while still delivering a good relevance. This scheme uses both lexical BM2…

Retrieval