paper-with-me

홈 › Papers

Silent versus modal multi-speaker speech recognition from ultrasound and video

2021-02-27 · Manuel Sam Ribeiro, Aciel Eshky, Korin Richmond, Steve Renals

We investigate multi-speaker speech recognition from ultrasound images of the tongue and video images of the lips. We train our systems on imaging data from modal speech, and evaluate on matched test sets of two speaking modes: silent and modal speech. We observe that silent speech recognition from imaging data underperforms compared to modal speech recognition, likely due to a speaking-mode mismatch between training and testing. We improve silent speech recognition performance using techniques that address the domain mismatch, such as fMLLR and unsupervised model adaptation. We also analyse the properties of silent and modal speech in terms of utterance duration and the size of the articulatory space. To estimate the articulatory space, we compute the convex hull of tongue splines, extracted from ultrasound tongue images. Overall, we observe that the duration of silent speech is longer than that of modal speech, and that silent speech covers a smaller articulatory space than modal speech. Although these two properties are statistically significant across speaking modes, they do not directly correlate with word error rates from speech recognition.

📄 PDF Abstract BibTeX arXiv:2103.00333

Code (0)

등록된 구현이 없습니다.

Tasks

Silent Speech Recognitionspeech-recognitionSpeech Recognition

Similar Papers 제목 키워드 기반

VCVTS: Multi-speaker Video-to-Speech synthesis via cross-modal knowledge transfer from voice conversion

2022-02-18 · Disong Wang, Shan Yang, Dan Su, Xunying Liu 외

Though significant progress has been made for speaker-dependent Video-to-Speech (VTS) synthesis, little attention is devoted to multi-speaker VTS that can map silent video to speech, while allowing flexible control of sp…

QuantizationSpeech SynthesisTransfer LearningVoice Conversion

MuteSwap: Visual-informed Silent Video Identity Conversion

2025-07-01 · Yifan Liu, Yu Fang, Zhouhan Lin arxiv

Conventional voice conversion modifies voice characteristics from a source speaker to a target speaker, relying on audio input from both sides. However, this process becomes infeasible when clean audio is unavailable, su…

Contrastive LearningVoice ConversionSpeech Synthesis

Cross-Modal Masking for Robust Silent Speech Synthesis Using sEMG and Lipreading

2026-06-08 · Eder del Blanco, David Gimeno-Gómez, Eva Navas, Carlos-D. Martínez-Hinarejos 외 arxiv

Speech restoration through silent speech interfaces (SSIs) has emerged as a promising assistive technology for individuals with impaired or absent laryngeal voice production. Among non-invasive SSI modalities, surface el…

Speech Synthesis

Multimodal Corpora for Silent Speech Interaction

2014-05-01 · LREC 2014 5 · Jo{\~a}o Freitas, Ant{\'o}nio Teixeira, Miguel Dias

A Silent Speech Interface (SSI) allows for speech communication to take place in the absence of an acoustic signal. This type of interface is an alternative to conventional Automatic Speech Recognition which is not adequ…

Automatic Speech RecognitionAutomatic Speech Recognition (ASR)speech-recognitionSpeech Recognition

VisageSynTalk: Unseen Speaker Video-to-Speech Synthesis via Speech-Visage Feature Selection

2022-06-15 · Joanna Hong, Minsu Kim, Yong Man Ro

The goal of this work is to reconstruct speech from a silent talking face video. Recent studies have shown impressive performance on synthesizing speech from silent talking face videos. However, they have not explicitly …

feature selectionSpeech Synthesis