DreamDiffusion: Generating High-Quality Images from Brain EEG Signals
This paper introduces DreamDiffusion, a novel method for generating high-quality images directly from brain electroencephalogram (EEG) signals, without the need to translate thoughts into text. DreamDiffusion leverages pre-trained text-to-image models and employs temporal masked signal modeling to pre-train the EEG encoder for effective and robust EEG representations. Additionally, the method further leverages the CLIP image encoder to provide extra supervision to better align EEG, text, and image embeddings with limited EEG-image pairs. Overall, the proposed method overcomes the challenges of using EEG signals for image generation, such as noise, limited information, and individual differences, and achieves promising results. Quantitative and qualitative results demonstrate the effectiveness of the proposed method as a significant step towards portable and low-cost ``thoughts-to-image'', with potential applications in neuroscience and computer vision. The code is available here \url{https://github.com/bbaaii/DreamDiffusion}.
Code (1)
Tasks
EEGElectroencephalogram (EEG)Image GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Novel Approach to Image EEG Sleep Data for Improving Quality of Life in Patients Suffering From Brain Injuries Using DreamDiffusion
Those experiencing strokes, traumatic brain injuries, and drug complications can often end up hospitalized and diagnosed with coma or locked-in syndrome. Such mental impediments can permanently alter the neurological pat…
EEGSYNAPSE: Synergizing an Adapter and Finetuning for High-Fidelity EEG Synthesis from a CLIP-Aligned Encoder
Recent progress in diffusion-based generative models has enabled high-quality image synthesis conditioned on diverse modalities. Extending such models to brain signals could deepen our understanding of human perception a…
Representation LearningImage GenerationBrainDreamer: Reasoning-Coherent and Controllable Image Generation from EEG Brain Signals via Language Guidance
Can we directly visualize what we imagine in our brain together with what we describe? The inherent nature of human perception reveals that, when we think, our body can combine language description and build a vivid pict…
Contrastive LearningEEGElectroencephalogram (EEG)Image GenerationNon-Reference Quality Assessment for Medical Imaging: Application to Synthetic Brain MRIs
Generating high-quality synthetic data is crucial for addressing challenges in medical imaging, such as domain adaptation, data scarcity, and privacy concerns. Existing image quality metrics often rely on reference image…
BenchmarkingDomain AdaptationDecoding Realistic Images from Brain Activity with Contrastive Self-supervision and Latent Diffusion
Reconstructing visual stimuli from human brain activities provides a promising opportunity to advance our understanding of the brain's visual system and its connection with computer vision models. Although deep generativ…
Contrastive Learning