LipLearner: Customizable Silent Speech Interactions on Mobile Devices
Silent speech interface is a promising technology that enables private communications in natural language. However, previous approaches only support a small and inflexible vocabulary, which leads to limited expressiveness. We leverage contrastive learning to learn efficient lipreading representations, enabling few-shot command customization with minimal user effort. Our model exhibits high robustness to different lighting, posture, and gesture conditions on an in-the-wild dataset. For 25-command classification, an F1-score of 0.8947 is achievable only using one shot, and its performance can be further boosted by adaptively learning from more data. This generalizability allowed us to develop a mobile silent speech interface empowered with on-device fine-tuning and visual keyword spotting. A user study demonstrated that with LipLearner, users could define their own commands with high reliability guaranteed by an online incremental learning scheme. Subjective feedback indicated that our system provides essential functionalities for customizable silent speech interactions with high usability and learnability.
Code (1)
Tasks
Contrastive LearningIncremental LearningKeyword SpottingLipreadingVisual Keyword SpottingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Watch Your Mouth: Silent Speech Recognition with Depth Sensing
Silent speech recognition is a promising technology that decodes human speech without requiring audio signals, enabling private human-computer interactions. In this paper, we propose Watch Your Mouth, a novel method that…
Deep LearningLipreadingSilent Speech Recognitionspeech-recognition+2Digital Voicing of Silent Speech
In this paper, we consider the task of digitally voicing silent speech, where silently mouthed words are converted to audible speech based on electromyography (EMG) sensor measurements that capture muscle impulses. While…
Electromyography (EMG)Speech SynthesisSilent versus modal multi-speaker speech recognition from ultrasound and video
We investigate multi-speaker speech recognition from ultrasound images of the tongue and video images of the lips. We train our systems on imaging data from modal speech, and evaluate on matched test sets of two speaking…
Silent Speech Recognitionspeech-recognitionSpeech RecognitionImproving the Gap in Visual Speech Recognition Between Normal and Silent Speech Based on Metric Learning
This paper presents a novel metric learning approach to address the performance gap between normal and silent speech in visual speech recognition (VSR). The difference in lip movements between the two poses a challenge f…
Metric Learningspeech-recognitionSpeech RecognitionVisual Speech RecognitionVisual-Only Recognition of Normal, Whispered and Silent Speech
Silent speech interfaces have been recently proposed as a way to enable communication when the acoustic signal is not available. This introduces the need to build visual speech recognition systems for silent and whispere…
Silent Speech Recognitionspeech-recognitionSpeech RecognitionVisual Speech Recognition