Improving Brain-to-Image Reconstruction via Fine-Grained Text Bridging
Brain-to-Image reconstruction aims to recover visual stimuli perceived by humans from brain activity. However, the reconstructed visual stimuli often missing details and semantic inconsistencies, which may be attributed to insufficient semantic information. To address this issue, we propose an approach named Fine-grained Brain-to-Image reconstruction (FgB2I), which employs fine-grained text as bridge to improve image reconstruction. FgB2I comprises three key stages: detail enhancement, decoding fine-grained text descriptions, and text-bridged brain-to-image reconstruction. In the detail-enhancement stage, we leverage large vision-language models to generate fine-grained captions for visual stimuli and experimentally validate its importance. We propose three reward metrics (object accuracy, text-image semantic similarity, and image-image semantic similarity) to guide the language model in decoding fine-grained text descriptions from fMRI signals. The fine-grained text descriptions can be integrated into existing reconstruction methods to achieve fine-grained Brain-to-Image reconstruction.
Code (0)
등록된 구현이 없습니다.
Tasks
Image ReconstructionLanguage ModelingLanguage ModellingSemantic SimilaritySemantic Textual SimilaritySimilar Papers 제목 키워드 기반
Decoding natural image stimuli from fMRI data with a surface-based convolutional network
Due to the low signal-to-noise ratio and limited resolution of functional MRI data, and the high complexity of natural images, reconstructing a visual stimulus from human brain fMRI measurements is a challenging task. In…
BrainVis: Exploring the Bridge between Brain and Visual Signals via Image Reconstruction
Analyzing and reconstructing visual stimuli from brain signals effectively advances the understanding of human visual system. However, the EEG signals are complex and contain significant noise. This leads to substantial …
cross-modal alignmentEEGImage ReconstructionReverse the auditory processing pathway: Coarse-to-fine audio reconstruction from fMRI
Drawing inspiration from the hierarchical processing of the human auditory system, which transforms sound from low-level acoustic features to high-level semantic understanding, we introduce a novel coarse-to-fine audio r…
FADVersatile Framework with Semantic and Structural guidance for Image Reconstruction from Brain Activity
Reconstructing visual stimuli from brain recordings has been a meaningful and challenging task in brain decoding. Especially, the achievement of precise and controllable image reconstruction bears great significance in p…
Text-to-Image GenerationImage ReconstructionBrain DecodingAREPAS: Anomaly Detection in Fine-Grained Anatomy with Reconstruction-Based Semantic Patch-Scoring
Early detection of newly emerging diseases, lesion severity assessment, differentiation of medical conditions and automated screening are examples for the wide applicability and importance of anomaly detection (AD) and u…
Image-to-Image TranslationLesion SegmentationAnomaly Detection