A Penny for Your Thoughts: Decoding Speech from Inexpensive Brain Signals
We explore whether neural networks can decode brain activity into speech by mapping EEG recordings to audio representations. Using EEG data recorded as subjects listened to natural speech, we train a model with a contrastive CLIP loss to align EEG-derived embeddings with embeddings from a pre-trained transformer-based speech model. Building on the state-of-the-art EEG decoder from Meta, we introduce three architectural modifications: (i) subject-specific attention layers (+0.15% WER improvement), (ii) personalized spatial attention (+0.45%), and (iii) a dual-path RNN with attention (-1.87%). Two of the three modifications improved performance, highlighting the promise of personalized architectures for brain-to-speech decoding and applications in brain-computer interfaces.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
A Penny for Your (visual) Thoughts: Self-Supervised Reconstruction of Natural Movies from Brain Activity
Reconstructing natural videos from fMRI brain recordings is very challenging, for two main reasons: (i) As fMRI data acquisition is difficult, we only have a limited amount of supervised samples, which is not enough to c…
DecoderSPEAK YOUR MIND! Towards Imagined Speech Recognition With Hierarchical Deep Learning
Speech-related Brain Computer Interface (BCI) technologies provide effective vocal communication strategies for controlling devices through speech commands interpreted from brain signals. In order to infer imagined speec…
Brain Computer InterfaceGeneral Classificationspeech-recognitionSpeech Recognition+1Magnetoencephalography (MEG) Based Non-Invasive Chinese Speech Decoding
As an emerging paradigm of brain-computer interfaces (BCIs), speech BCI has the potential to directly reflect auditory perception and thoughts, offering a promising communication alternative for patients with aphasia. Ch…
Mitigating Noisy Inputs for Question Answering
Natural language processing systems are often downstream of unreliable inputs: machine translation, optical character recognition, or speech recognition. For instance, virtual assistants can only answer your questions af…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Machine TranslationOptical Character Recognition+5Deep Learning the EEG Manifold for Phonological Categorization from Active Thoughts
Speech-related Brain Computer Interfaces (BCI) aim primarily at finding an alternative vocal communication pathway for people with speaking disabilities. As a step towards full decoding of imagined speech from active tho…
Binary ClassificationDeep LearningEEGElectroencephalogram (EEG)+2