DeepSuM: Deep Sufficient Modality Learning Framework
Multimodal learning has become a pivotal approach in developing robust learning models with applications spanning multimedia, robotics, large language models, and healthcare. The efficiency of multimodal systems is a critical concern, given the varying costs and resource demands of different modalities. This underscores the necessity for effective modality selection to balance performance gains against resource expenditures. In this study, we propose a novel framework for modality selection that independently learns the representation of each modality. This approach allows for the assessment of each modality's significance within its unique representation space, enabling the development of tailored encoders and facilitating the joint analysis of modalities with distinct characteristics. Our framework aims to enhance the efficiency and effectiveness of multimodal learning by optimizing modality integration and selection.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
DeepSUM++: Non-local Deep Neural Network for Super-Resolution of Unregistered Multitemporal Images
Deep learning methods for super-resolution of a remote sensing scene from multiple unregistered low-resolution images have recently gained attention thanks to a challenge proposed by the European Space Agency. This paper…
Super-ResolutionDeepSUM: Deep neural network for Super-resolution of Unregistered Multitemporal images
Recently, convolutional neural networks (CNN) have been successfully applied to many remote sensing problems. However, deep learning techniques for multi-image super-resolution from multitemporal unregistered imagery hav…
Image Super-ResolutionMulti-Frame Super-ResolutionRepresentation LearningSuper-ResolutionDisentangling and Generating Modalities for Recommendation in Missing Modality Scenarios
Multi-modal recommender systems (MRSs) have achieved notable success in improving personalization by leveraging diverse modalities such as images, text, and audio. However, two key challenges remain insufficiently addres…
Cross-Modal RetrievalRecommendation SystemsCross-Modality Deep Feature Learning for Brain Tumor Segmentation
Recent advances in machine learning and prevalence of digital medical images have opened up an opportunity to address the challenging brain tumor segmentation (BTS) task by using deep convolutional neural networks. Howev…
Brain Tumor SegmentationSegmentationTumor SegmentationA Unified Conditional Disentanglement Framework for Multimodal Brain MR Image Translation
Multimodal MRI provides complementary and clinically relevant information to probe tissue condition and to characterize various diseases. However, it is often difficult to acquire sufficiently many modalities from the sa…
DecoderDisentanglementTranslationTumor Segmentation