Multistream neural architectures for cued-speech recognition using a pre-trained visual feature extractor and constrained CTC decoding
This paper proposes a simple and effective approach for automatic recognition of Cued Speech (CS), a visual communication tool that helps people with hearing impairment to understand spoken language with the help of hand gestures that can uniquely identify the uttered phonemes in complement to lipreading. The proposed approach is based on a pre-trained hand and lips tracker used for visual feature extraction and a phonetic decoder based on a multistream recurrent neural network trained with connectionist temporal classification loss and combined with a pronunciation lexicon. The proposed system is evaluated on an updated version of the French CS dataset CSF18 for which the phonetic transcription has been manually checked and corrected. With a decoding accuracy at the phonetic level of 70.88%, the proposed system outperforms our previous CNN-HMM decoder and competes with more complex baselines.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderLipreadingspeech-recognitionSpeech RecognitionSimilar Papers 제목 키워드 기반
ASAPP-ASR: Multistream CNN and Self-Attentive SRU for SOTA Speech Recognition
In this paper we present state-of-the-art (SOTA) performance on the LibriSpeech corpus with two novel neural network architectures, a multistream CNN for acoustic modeling and a self-attentive simple recurrent unit (SRU)…
Data AugmentationDiversityLanguage ModelingLanguage Modelling+2Cued Speech Generation Leveraging a Pre-trained Audiovisual Text-to-Speech Model
This paper presents a novel approach for the automatic generation of Cued Speech (ACSG), a visual communication system used by people with hearing impairment to better elicit the spoken language. We explore transfer lear…
text-to-speechText to SpeechTransfer LearningMultistream CNN for Robust Acoustic Modeling
This paper proposes multistream CNN, a novel neural network architecture for robust acoustic modeling in speech recognition tasks. The proposed architecture processes input speech with diverse temporal resolutions by app…
Data Augmentationspeech-recognitionSpeech RecognitionCued-Agent: A Collaborative Multi-Agent System for Automatic Cued Speech Recognition
Cued Speech (CS) is a visual communication system that combines lip-reading with hand coding to facilitate communication for individuals with hearing impairments. Automatic CS Recognition (ACSR) aims to convert CS hand g…
Speech RecognitionCross-Modal Mutual Learning for Cued Speech Recognition
Automatic Cued Speech Recognition (ACSR) provides an intelligent human-machine interface for visual communications, where the Cued Speech (CS) system utilizes lip movements and hand gestures to code spoken language for h…
speech-recognitionSpeech Recognition