Learning Mutual Modulation for Self-Supervised Cross-Modal Super-Resolution
Self-supervised cross-modal super-resolution (SR) can overcome the difficulty of acquiring paired training data, but is challenging because only low-resolution (LR) source and high-resolution (HR) guide images from different modalities are available. Existing methods utilize pseudo or weak supervision in LR space and thus deliver results that are blurry or not faithful to the source modality. To address this issue, we present a mutual modulation SR (MMSR) model, which tackles the task by a mutual modulation strategy, including a source-to-guide modulation and a guide-to-source modulation. In these modulations, we develop cross-domain adaptive filters to fully exploit cross-modal spatial dependency and help induce the source to emulate the resolution of the guide and induce the guide to mimic the modality characteristics of the source. Moreover, we adopt a cycle consistency constraint to train MMSR in a fully self-supervised manner. Experiments on various tasks demonstrate the state-of-the-art performance of our MMSR.
Code (1)
Tasks
Super-ResolutionSimilar Papers 제목 키워드 기반
Self-MI: Efficient Multimodal Fusion via Self-Supervised Multi-Task Learning with Auxiliary Mutual Information Maximization
Multimodal representation learning poses significant challenges in capturing informative and distinct features from multiple modalities. Existing methods often struggle to exploit the unique characteristics of each modal…
Multi-Task LearningRepresentation LearningSelf-Supervised LearningCMD: Self-supervised 3D Action Representation Learning with Cross-modal Mutual Distillation
In 3D action recognition, there exists rich complementary information between skeleton modalities. Nevertheless, how to model and utilize this information remains a challenging problem for self-supervised 3D action repre…
3D Action RecognitionAction RecognitionFew-Shot Skeleton-Based Action RecognitionKnowledge Distillation+2TVDIM: Enhancing Image Self-Supervised Pretraining via Noisy Text Data
Among ubiquitous multimodal data in the real world, text is the modality generated by human, while image reflects the physical world honestly. In a visual understanding application, machines are expected to understand im…
Contrastive Learningimage-classificationImage ClassificationSelf-Supervised LearningA Novel Self-Supervised Cross-Modal Image Retrieval Method In Remote Sensing
Due to the availability of multi-modal remote sensing (RS) image archives, one of the most important research topics is the development of cross-modal RS image retrieval (CM-RSIR) methods that search semantically similar…
Image RetrievalRetrievalLayer-Specific Lipschitz Modulation for Fault-Tolerant Multimodal Representation Learning
Modern multimodal systems deployed in industrial and safety-critical environments must remain reliable under partial sensor failures, signal degradation, or cross-modal inconsistencies. This work introduces a mathematica…
Self-Supervised Anomaly DetectionRepresentation Learning