TS-Net: Combining modality specific and common features for multimodal patch matching
Multimodal patch matching addresses the problem of finding the correspondences between image patches from two different modalities, e.g. RGB vs sketch or RGB vs near-infrared. The comparison of patches of different modalities can be done by discovering the information common to both modalities (Siamese like approaches) or the modality-specific information (Pseudo-Siamese like approaches). We observed that none of these two scenarios is optimal. This motivates us to propose a three-stream architecture, dubbed as TS-Net, combining the benefits of the two. In addition, we show that adding extra constraints in the intermediate layers of such networks further boosts the performance. Experimentations on three multimodal datasets show significant performance gains in comparison with Siamese and Pseudo-Siamese networks.
Code (1)
Tasks
Multimodal Patch MatchingPatch MatchingSimilar Papers 제목 키워드 기반
Adversarial Multimodal Representation Learning for Click-Through Rate Prediction
For better user experience and business effectiveness, Click-Through Rate (CTR) prediction has been one of the most important tasks in E-commerce. Although extensive CTR prediction models have been proposed, learning goo…
Click-Through Rate PredictionPredictionRepresentation LearningProgressive Modality Reinforcement for Human Multimodal Emotion Recognition From Unaligned Multimodal Sequences
Human multimodal emotion recognition involves time-series data of different modalities, such as natural language, visual motions, and acoustic behaviors. Due to the variable sampling rates for sequences from differen…
Emotion RecognitionMultimodal Emotion RecognitionTime SeriesTime Series AnalysisCentralNet: a Multilayer Approach for Multimodal Fusion
This paper proposes a novel multimodal fusion approach, aiming to produce best possible decisions by integrating information coming from multiple media. While most of the past multimodal approaches either work by project…
Multi-Task LearningMulti-Modality Collaborative Learning for Sentiment Analysis
Multimodal sentiment analysis (MSA) identifies individuals' sentiment states in videos by integrating visual, audio, and text modalities. Despite progress in existing methods, the inherent modality heterogeneity limits t…
Multimodal Sentiment AnalysisSentiment AnalysisPrior-guided Fusion of Multimodal Features for Change Detection from Optical-SAR Images
Multimodal change detection (MMCD) identifies changed areas in multimodal remote sensing data, demonstrating significant application value in land use monitoring and urban sustainable development. However, literature MMC…
Feature ImportanceChange Detection