Modulate and Reconstruct: Learning Hyperspectral Imaging from Misaligned Smartphone Views
Hyperspectral reconstruction (HSR) from RGB images is a highly promising direction for accurate color reproduction and material color measurement. While most existing approaches rely on a single RGB image - thereby limiting reconstruction accuracy - the majority of modern smartphones are equipped with two or more cameras. In this work, we propose a novel multi-image-to-hyperspectral reconstruction (MI-HSR) framework that leverages a triple-camera smartphone system, where two lenses are equipped with carefully selected spectral filters. Our easy-to-implement configuration, based on theoretical and empirical analysis, allows to obtain more complete and diverse spectral data than traditional single-chamber setups. To support this new paradigm, we introduce Doomer, the first dataset for MI-HSR, comprising aligned images from three smartphone cameras and a hyperspectral reference camera across diverse scenes. We further introduce a lightweight alignment module for MI-HSR that effectively fuses multi-view inputs while mitigating parallax- and occlusion-induced artifacts. Proposed module demonstrate consistent quality improvements for modern HSR methods. In a nutshell, our setup allows 30% more accurate estimations of spectra compared to an ordinary RGB camera, while the proposed alignment module boosts the reconstruction quality of SotA methods by an additional 5%. Our findings suggest that spectral filtering of multiple views with commodity hardware unlocks more accurate and practical hyperspectral imaging.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Hyperspectral image reconstruction by deep learning with super-Rayleigh speckles
Ghost imaging via sparsity constraints (GISC) spectral camera modulates the three-dimensional (3D) hyperspectral image into a two-dimensional (2D) compressive image with speckles in a single shot. It obtains a 3D hypersp…
Deep LearningImage ReconstructionmHealth hyperspectral learning for instantaneous spatiospectral imaging of hemodynamics
Hyperspectral imaging acquires data in both the spatial and frequency domains to offer abundant physical or biological information. However, conventional hyperspectral imaging has intrinsic limitations of bulky instrumen…
compressed sensingRobust Image Reconstruction with Misaligned Structural Information
Multi-modality (or multi-channel) imaging is becoming increasingly important and more widely available, e.g. hyperspectral imaging in remote sensing, spectral CT in material sciences as well as multi-contrast MRI and PET…
Image ReconstructionFUN: A Focal U-Net Combining Reconstruction and Object Detection for Snapshot Spectral Imaging
Conventional push-broom hyperspectral imaging suffers from slow acquisition speeds, precluding real-time object detection; in contrast, snapshot spectral imaging enables instantaneous hyperspectral images (HSIs) capture,…
Real-Time Object DetectionMulti-Task LearningProximal Gradient Descent Unfolding Dense-spatial Spectral-attention Transformer for Compressive Spectral Imaging
The Coded Aperture Snapshot Spectral Compressive Imaging (CASSI) system modulates three-dimensional hyperspectral images into two-dimensional compressed images in a single exposure. Subsequently, three-dimensional hypers…
GPU