Towards RGB-NIR Cross-modality Image Registration and Beyond
This paper focuses on the area of RGB(visible)-NIR(near-infrared) cross-modality image registration, which is crucial for many downstream vision tasks to fully leverage the complementary information present in visible and infrared images. In this field, researchers face two primary challenges - the absence of a correctly-annotated benchmark with viewpoint variations for evaluating RGB-NIR cross-modality registration methods and the problem of inconsistent local features caused by the appearance discrepancy between RGB-NIR cross-modality images. To address these challenges, we first present the RGB-NIR Image Registration (RGB-NIR-IRegis) benchmark, which, for the first time, enables fair and comprehensive evaluations for the task of RGB-NIR cross-modality image registration. Evaluations of previous methods highlight the significant challenges posed by our RGB-NIR-IRegis benchmark, especially on RGB-NIR image pairs with viewpoint variations. To analyze the causes of the unsatisfying performance, we then design several metrics to reveal the toxic impact of inconsistent local features between visible and infrared images on the model performance. This further motivates us to develop a baseline method named Semantic Guidance Transformer (SGFormer), which utilizes high-level semantic guidance to mitigate the negative impact of local inconsistent features. Despite the simplicity of our motivation, extensive experimental results show the effectiveness of our method.
Code (0)
등록된 구현이 없습니다.
Tasks
Image RegistrationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
KLDivNet: An unsupervised neural network for multi-modality image registration
Multi-modality image registration is one of the most underlined processes in medical image analysis. Recently, convolutional neural networks (CNNs) have shown significant potential in deformable registration. However, th…
Image RegistrationMedical Image AnalysisMedical Image RegistrationDeep Learning based Inter-Modality Image Registration Supervised by Intra-Modality Similarity
Non-rigid inter-modality registration can facilitate accurate information fusion from different modalities, but it is challenging due to the very different image appearances across modalities. In this paper, we propose t…
Image RegistrationCross-Modality Image Registration using a Training-Time Privileged Third Modality
In this work, we consider the task of pairwise cross-modality image registration, which may benefit from exploiting additional images available only at training time from an additional modality that is different to those…
Image RegistrationUnsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration
Recent learning-based image fusion methods have marked numerous progress in pre-registered multi-modality data, but suffered serious ghosts dealing with misaligned multi-modality data, due to the spatial deformation and …
Image GenerationInfrared And Visible Image FusionStyle TransferUnsupervised Image Registration Towards Enhancing Performance and Explainability in Cardiac And Brain Image Analysis
Magnetic Resonance Imaging (MRI) typically recruits multiple sequences (defined here as "modalities"). As each modality is designed to offer different anatomical and functional clinical information, there are evident dis…
Image GenerationImage RegistrationMedical Image AnalysisUnsupervised Image Registration