Multi-modal Speech Enhancement with Limited Electromyography Channels
Speech enhancement (SE) aims to improve the clarity, intelligibility, and quality of speech signals for various speech enabled applications. However, air-conducted (AC) speech is highly susceptible to ambient noise, particularly in low signal-to-noise ratio (SNR) and non-stationary noise environments. Incorporating multi-modal information has shown promise in enhancing speech in such challenging scenarios. Electromyography (EMG) signals, which capture muscle activity during speech production, offer noise-resistant properties beneficial for SE in adverse conditions. Most previous EMG-based SE methods required 35 EMG channels, limiting their practicality. To address this, we propose a novel method that considers only 8-channel EMG signals with acoustic signals using a modified SEMamba network with added cross-modality modules. Our experiments demonstrate substantial improvements in speech quality and intelligibility over traditional approaches, especially in extremely low SNR settings. Notably, compared to the SE (AC) approach, our method achieves a significant PESQ gain of 0.235 under matched low SNR conditions and 0.527 under mismatched conditions, highlighting its robustness.
Code (0)
등록된 구현이 없습니다.
Tasks
Electromyography (EMG)Speech EnhancementSimilar Papers 제목 키워드 기반
EMGSE: Acoustic/EMG Fusion for Multimodal Speech Enhancement
Multimodal learning has been proven to be an effective method to improve speech enhancement (SE) performance, especially in challenging situations such as low signal-to-noise ratios, speech noise, or unseen noise types. …
Electromyography (EMG)Speech EnhancementDeep Speech Synthesis from Multimodal Articulatory Representations
The amount of articulatory data available for training deep learning models is much less compared to acoustic speech data. In order to improve articulatory-to-acoustic synthesis performance in these low-resource settings…
Speech SynthesisTransfer LearningBone-conduction Guided Multimodal Speech Enhancement with Conditional Diffusion Models
Single-channel speech enhancement models face significant performance degradation in extremely noisy environments. While prior work has shown that complementary bone-conducted speech can guide enhancement, effective inte…
Speech EnhancementA Study of Incorporating Articulatory Movement Information in Speech Enhancement
Although deep learning algorithms are widely used for improving speech enhancement (SE) performance, the performance remains limited under highly challenging conditions, such as unseen noise or noise signals having low s…
Speech EnhancementCross-Modal Masking for Robust Silent Speech Synthesis Using sEMG and Lipreading
Speech restoration through silent speech interfaces (SSIs) has emerged as a promising assistive technology for individuals with impaired or absent laryngeal voice production. Among non-invasive SSI modalities, surface el…
Speech Synthesis