Multimodal sparse representation learning and applications

2015-11-19 · Miriam Cha, Youngjune Gwon, H. T. Kung

Unsupervised methods have proven effective for discriminative tasks in a single-modality scenario. In this paper, we present a multimodal framework for learning sparse representations that can capture semantic correlation between modalities. The framework can model relationships at a higher level by forcing the shared sparse representation. In particular, we propose the use of joint dictionary learning technique for sparse coding and formulate the joint representation for concision, cross-modal representations (in case of a missing modality), and union of the cross-modal representations. Given the accelerated growth of multimodal data posted on the Web such as YouTube, Wikipedia, and Twitter, learning good multimodal features is becoming increasingly important. We show that the shared representations enabled by our framework substantially improve the classification performance under both unimodal and multimodal settings. We further show how deep architectures built on the proposed framework are effective for the case of highly nonlinear correlations between modalities. The effectiveness of our approach is demonstrated experimentally in image denoising, multimedia event detection and retrieval on the TRECVID dataset (audio-video), category classification on the Wikipedia dataset (image-text), and sentiment classification on PhotoTweet (image-text).

📄 PDF Abstract BibTeX arXiv:1511.06238

Code (0)

등록된 구현이 없습니다.

Tasks

Classification DenoisingDictionary LearningEvent DetectionGeneral Classification Image Denoising Representation Learning Retrieval Sentiment AnalysisSentiment Classification

UEmbed: Unified Sparse and Dense Multimodal Embeddings

2026-08-03 · Tingyu Song, Mingxin Li, Yanzhao Zhang, Dingkun Long 외 hf

Sparse retrieval underpins modern search systems, from web search to retrieval-augmented generation. Existing work has introduced Learned Sparse Retrieval (LSR) to push beyond exact lexical matching toward richer semanti…

Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)

2024-02-16 · Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio P. Calmon 외

CLIP embeddings have demonstrated remarkable performance across a wide range of multimodal applications. However, these high-dimensional, dense vector representations are not easily interpretable, limiting our understand…

Model Editing

Multimodal Sparse Coding for Event Detection

2016-05-17 · Youngjune Gwon, William Campbell, Kevin Brady, Douglas Sturim 외

Unsupervised feature learning methods have proven effective for classification tasks based on a single modality. We present multimodal sparse coding for learning feature representations shared across multiple modalities.…

Classification Event Detection General Classification

Multimodal Task-Driven Dictionary Learning for Image Classification

2015-02-04 · Soheil Bahrampour, Nasser M. Nasrabadi, Asok Ray, W. Kenneth Jenkins

Dictionary learning algorithms have been successfully used for both reconstructive and discriminative tasks, where an input signal is represented with a sparse linear combination of dictionary atoms. While these methods …

Action Recognition Classification Dictionary Learning Face Recognition+4

Online Convolutional Dictionary Learning for Multimodal Imaging

2017-06-13 · Kevin Degraux, Ulugbek S. Kamilov, Petros T. Boufounos, Dehong Liu

Computational imaging methods that can exploit multiple modalities have the potential to enhance the capabilities of traditional sensing systems. In this paper, we propose a new method that reconstructs multimodal images…

Dictionary Learning