An Improved Model for Voicing Silent Speech

2021-06-03 · ACL 2021 5 · David Gaddy, Dan Klein

In this paper, we present an improved model for voicing silent speech, where audio is synthesized from facial electromyography (EMG) signals. To give our model greater flexibility to learn its own input features, we directly use EMG signals as input in the place of hand-designed features used by prior work. Our model uses convolutional layers to extract features from the signals and Transformer layers to propagate information across longer distances. To provide better signal for learning, we also introduce an auxiliary task of predicting phoneme labels in addition to predicting speech audio features. On an open vocabulary intelligibility evaluation, our model improves the state of the art for this task by an absolute 25.8%.

📄 PDF Abstract BibTeX arXiv:2106.01933

Code (1)

dgaddy/silent_speech 공식 구현 pytorch

Tasks

Electromyography (EMG)model

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Adam 설명 없음

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Digital Voicing of Silent Speech

2020-10-06 · EMNLP 2020 11 · David Gaddy, Dan Klein

In this paper, we consider the task of digitally voicing silent speech, where silently mouthed words are converted to audible speech based on electromyography (EMG) sensor measurements that capture muscle impulses. While…

Electromyography (EMG)Speech Synthesis

The Use of Voice Source Features for Sung Speech Recognition

2021-02-20 · Gerardo Roa Dabike, Jon Barker

In this paper, we ask whether vocal source features (pitch, shimmer, jitter, etc) can improve the performance of automatic sung speech recognition, arguing that conclusions previously drawn from spoken speech studies may…

speech-recognition Speech Recognition valid

A Silent Speech Decoding System from EEG and EMG with Heterogenous Electrode Configurations

2025-06-16 · Masakazu Inoue, Motoshige Sato, Kenichi Tomeoka, Nathania Nah 외

Silent speech decoding, which performs unvocalized human speech recognition from electroencephalography/electromyography (EEG/EMG), increases accessibility for speech-impaired humans. However, data collection is difficul…

EEG speech-recognition Speech Recognition

Improved Speech Reconstruction from Silent Video

2017-08-01 · Ariel Ephrat, Tavi Halperin, Shmuel Peleg

Speechreading is the task of inferring phonetic information from visually observed articulatory facial movements, and is a notoriously difficult task for humans to perform. In this paper we present an end-to-end model ba…

Lenition and Fortition of Stop Codas in Romanian

2020-05-01 · LREC 2020 5 · Mathilde Hutin, Oana Niculescu, Ioana Vasilescu, Lori Lamel 외

The present paper aims at providing a first study of lenition- and fortition-type phenomena in coda position in Romanian, a language that can be considered as less-resourced. Our data show that there are two contexts for…