Hyperrealistic neural decoding: Reconstruction of face stimuli from fMRI measurements via the GAN latent space
We introduce a new framework for hyperrealistic reconstruction of perceived naturalistic stimuli from brain recordings. To this end, we embrace the use of generative adversarial networks (GANs) at the earliest step of our neural decoding pipeline by acquiring functional magnetic resonance imaging data as subjects perceived face images created by the generator network of a GAN. Subsequently, we used a decoding approach to predict the latent state of the GAN from brain data. Hence, latent representations for stimulus (re-)generation are obtained, leading to state-of-the-art image reconstructions. Altogether, we have developed a highly promising approach for decoding sensory perception from brain activity and systematically analyzing neural information processing in the human brain.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
A Survey on fMRI-based Brain Decoding for Reconstructing Multimodal Stimuli
In daily life, we encounter diverse external stimuli, such as images, sounds, and videos. As research in multimodal stimuli and neuroscience advances, fMRI-based brain decoding has become a key tool for understanding bra…
Brain DecodingImage GenerationSurveyUniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language
Decoding text stimuli from cognitive signals (e.g. fMRI) enhances our understanding of the human language system, paving the way for building versatile Brain-Computer Interface. However, existing studies largely focus on…
Brain Computer InterfaceBrain DecodingDecoderEEG+3fMRI2GES: Co-speech Gesture Reconstruction from fMRI Signal with Dual Brain Decoding Alignment
Understanding how the brain responds to external stimuli and decoding this process has been a significant challenge in neuroscience. While previous studies typically concentrated on brain-to-image and brain-to-language r…
Brain DecodingAccurate reconstruction of image stimuli from human fMRI based on the decoding model with capsule network architecture
In neuroscience, all kinds of computation models were designed to answer the open question of how sensory stimuli are encoded by neurons and conversely, how sensory stimuli can be decoded from neuronal activities. Especi…
Open-Ended Question AnsweringSSIMFrom Flat to Round: Redefining Brain Decoding with Surface-Based fMRI and Cortex Structure
Reconstructing visual stimuli from human brain activity (e.g., fMRI) bridges neuroscience and computer vision by decoding neural representations. However, existing methods often overlook critical brain structure-function…
Brain Decoding