Joint fMRI Decoding and Encoding with Latent Embedding Alignment
The connection between brain activity and corresponding visual stimuli is crucial in comprehending the human brain. While deep generative models have exhibited advancement in recovering brain recordings by generating images conditioned on fMRI signals, accomplishing high-quality generation with consistent semantics continues to pose challenges. Moreover, the prediction of brain activity from visual stimuli remains a formidable undertaking. In this paper, we introduce a unified framework that addresses both fMRI decoding and encoding. Commencing with the establishment of two latent spaces capable of representing and reconstructing fMRI signals and visual images, respectively, we proceed to align the fMRI signals and visual images within the latent space, thereby enabling a bidirectional transformation between the two domains. Our Latent Embedding Alignment (LEA) model concurrently recovers visual stimuli from fMRI signals and predicts brain activity from images within a unified framework. The performance of LEA surpasses that of existing methods on multiple benchmark fMRI decoding and encoding datasets. By integrating fMRI decoding and encoding, LEA offers a comprehensive solution for modeling the intricate relationship between brain activity and visual stimuli.
Code (0)
등록된 구현이 없습니다.
Tasks
Image GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Statistical Learning for Latent Embedding Alignment with Application to Brain Encoding and Decoding
Brain encoding and decoding aims to understand the relationship between external stimuli and brain activities, and is a fundamental problem in neuroscience. In this article, we study latent embedding alignment for brain …
Image ReconstructionTransfer LearningNonlocal operator learning for fMRI encoding and decoding tasks
Functional MRI data exhibit high-dimensional spatiotemporal structure, making both prediction and decoding challenging. In this work, we investigate neural integral-operator-based models for encoding and decoding tasks i…
Representation LearningfMRI Semantic Category Decoding using Linguistic Encoding of Word Embeddings
The dispute of how the human brain represents conceptual knowledge has been argued in many scientific fields. Brain imaging studies have shown that the spatial patterns of neural activation in the brain are correlated wi…
Word EmbeddingsLatent neural source recovery via transcoding of simultaneous EEG-fMRI
Simultaneous EEG-fMRI is a multi-modal neuroimaging technique that provides complementary spatial and temporal resolution for inferring a latent source space of neural activity. In this paper we address this inference pr…
EEGElectroencephalogram (EEG)Looking through the mind's eye via multimodal encoder-decoder networks
In this work, we explore the decoding of mental imagery from subjects using their fMRI measurements. In order to achieve this decoding, we first created a mapping between a subject's fMRI signals elicited by the videos t…
Decoder