Learning Sparse Disentangled Representations for Multimodal Exclusion Retrieval

2025-04-04 · Prachi, Sumit Bhatia, Srikanta Bedathur

Multimodal representations are essential for cross-modal retrieval, but they often lack interpretability, making it difficult to understand the reasoning behind retrieved results. Sparse disentangled representations offer a promising solution; however, existing methods rely heavily on text tokens, resulting in high-dimensional embeddings. In this work, we propose a novel approach that generates compact, fixed-size embeddings that maintain disentanglement while providing greater control over retrieval tasks. We evaluate our method on challenging exclusion queries using the MSCOCO and Conceptual Captions benchmarks, demonstrating notable improvements over dense models like CLIP, BLIP, and VISTA (with gains of up to 11% in AP@10), as well as over sparse disentangled models like VDR (achieving up to 21% gains in AP@10). Furthermore, we present qualitative results that emphasize the enhanced interpretability of our disentangled representations.

📄 PDF Abstract BibTeX arXiv:2504.03184

Code (0)

등록된 구현이 없습니다.

Tasks

Cross-Modal Retrieval Disentanglement Retrieval

Methods 이 논문이 사용한 방법론

CLIP Contrastive Language-Image Pre-training (CLIP), consisting of a simplified version of ConVIRT trained from scratch, is an efficient method of image representation learning…

BLIP Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based…

MoToRec: Sparse-Regularized Multimodal Tokenization for Cold-Start Recommendation

2026-02-11 · Jialin Liu, Zhaorui Zhang, Ray C. C. Cheung arxiv

Graph neural networks (GNNs) have revolutionized recommender systems by effectively modeling complex user-item interactions, yet data sparsity and the item cold-start problem significantly impair performance, particularl…

Multimodal Recommendation

DEO: Training-Free Direct Embedding Optimization for Negation-Aware Retrieval

2026-03-10 · Taegyeong Lee, Jiwon Park, Seunghyun Hwang, JooYoung Jang arxiv

Recent advances in Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) have enabled diverse retrieval methods. However, existing retrieval methods often fail to accurately retrieve results for negation …

SEM: Sparse Embedding Modulation for Post-Hoc Debiasing of Vision-Language Models

2026-03-19 · Quentin Guimard, Federico Bartsch, Simone Caldarella, Rahaf Aljundi 외 arxiv

Models that bridge vision and language, such as CLIP, are key components of multimodal AI, yet their large-scale, uncurated training data introduce severe social and spurious biases. Existing post-hoc debiasing methods o…

CADMR: Cross-Attention and Disentangled Learning for Multimodal Recommender Systems

2024-12-03 · Yasser Khalafaoui, Martino Lovisetto, Basarab Matei, Nistor Grozavu

The increasing availability and diversity of multimodal data in recommender systems offer new avenues for enhancing recommendation accuracy and user satisfaction. However, these systems must contend with high-dimensional…

Diversity Recommendation Systems

An Information Criterion for Controlled Disentanglement of Multimodal Data

2024-10-31 · Chenyu Wang, Sharut Gupta, Xinyi Zhang, Sana Tonekaboni 외

Multimodal representation learning seeks to relate and decompose information inherent in multiple modalities. By disentangling modality-specific information from information that is shared across modalities, we can impro…

counterfactual Disentanglement Representation Learning Self-Supervised Learning