Multimodal Deep Unfolding for Guided Image Super-Resolution
The reconstruction of a high resolution image given a low resolution observation is an ill-posed inverse problem in imaging. Deep learning methods rely on training data to learn an end-to-end mapping from a low-resolution input to a high-resolution output. Unlike existing deep multimodal models that do not incorporate domain knowledge about the problem, we propose a multimodal deep learning design that incorporates sparse priors and allows the effective integration of information from another image modality into the network architecture. Our solution relies on a novel deep unfolding operator, performing steps similar to an iterative algorithm for convolutional sparse coding with side information; therefore, the proposed neural network is interpretable by design. The deep unfolding architecture is used as a core component of a multimodal framework for guided image super-resolution. An alternative multimodal design is investigated by employing residual learning to improve the training efficiency. The presented multimodal approach is applied to super-resolution of near-infrared and multi-spectral images as well as depth upsampling using RGB images as side information. Experimental results show that our model outperforms state-of-the-art methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Image Super-ResolutionMultimodal Deep LearningSuper-ResolutionSimilar Papers 제목 키워드 기반
Multimodal Image Super-resolution via Deep Unfolding with Side Information
Deep learning methods have been successfully applied to various computer vision tasks. However, existing neural network architectures do not per se incorporate domain knowledge about the addressed problem, thus, understa…
Image Super-ResolutionSuper-ResolutionModel-Guided Multi-Contrast Deep Unfolding Network for MRI Super-resolution Reconstruction
Magnetic resonance imaging (MRI) with high resolution (HR) provides more detailed information for accurate diagnosis and quantitative image analysis. Despite the significant advances, most existing super-resolution (SR) …
Super-ResolutionAccurate and Lightweight Image Super-Resolution with Model-Guided Deep Unfolding Network
Deep neural networks (DNNs) based methods have achieved great success in single image super-resolution (SISR). However, existing state-of-the-art SISR techniques are designed like black boxes lacking transparency and int…
DenoisingImage Super-ResolutionSuper-ResolutionMemory-augmented Deep Unfolding Network for Guided Image Super-resolution
Guided image super-resolution (GISR) aims to obtain a high-resolution (HR) target image by enhancing the spatial resolution of a low-resolution (LR) target image under the guidance of a HR image. However, previous model-…
Image Super-ResolutionSuper-ResolutionInterpretable Deep Multimodal Image Super-Resolution
Multimodal image super-resolution (SR) is the reconstruction of a high resolution image given a low-resolution observation with the aid of another image modality. While existing deep multimodal models do not incorporate …
Image Super-ResolutionSuper-Resolution