Multimodal Learned Sparse Retrieval for Image Suggestion
Learned Sparse Retrieval (LSR) is a group of neural methods designed to encode queries and documents into sparse lexical vectors. These vectors can be efficiently indexed and retrieved using an inverted index. While LSR has shown promise in text retrieval, its potential in multi-modal retrieval remains largely unexplored. Motivated by this, in this work, we explore the application of LSR in the multi-modal domain, i.e., we focus on Multi-Modal Learned Sparse Retrieval (MLSR). We conduct experiments using several MLSR model configurations and evaluate the performance on the image suggestion task. We find that solving the task solely based on the image content is challenging. Enriching the image content with its caption improves the model performance significantly, implying the importance of image captions to provide fine-grained concepts and context information of images. Our approach presents a practical and effective solution for training LSR retrieval models in multi-modal settings.
Code (0)
등록된 구현이 없습니다.
Tasks
Image CaptioningRetrievalText RetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control
Learned sparse retrieval (LSR) is a family of neural methods that encode queries and documents into sparse lexical vectors that can be indexed and retrieved efficiently with an inverted index. We explore the application …
GPUImage RetrievalRetrievalText RetrievalMultimodal Query Suggestion with Multi-Agent Reinforcement Learning from Human Feedback
In the rapidly evolving landscape of information retrieval, search engines strive to provide more personalized and relevant results to users. Query suggestion systems play a crucial role in achieving this goal by assisti…
DiversityInformation RetrievalMulti-agent Reinforcement Learningreinforcement-learning+1UEmbed: Unified Sparse and Dense Multimodal Embeddings
Sparse retrieval underpins modern search systems, from web search to retrieval-augmented generation. Existing work has introduced Learned Sparse Retrieval (LSR) to push beyond exact lexical matching toward richer semanti…
Inference-Free Multimodal Learned Sparse Retrieval for Production-Scale Visual Document Search
As large-scale visual-document corpora such as arXiv papers and enterprise PDFs continue to grow, visual-document retrieval has gained increasing attention; yet it still lacks a deployable system that lexically indexes v…
Sparse and Dense Retrievers Learn Better Together: Joint Sparse-Dense Optimization for Text-Image Retrieval
Vision-Language Pretrained (VLP) models have achieved impressive performance on multimodal tasks, including text-image retrieval, based on dense representations. Meanwhile, Learned Sparse Retrieval (LSR) has gained tract…
Knowledge DistillationImage Retrieval