Combining Multiple Views for Visual Speech Recognition
Visual speech recognition is a challenging research problem with a particular practical application of aiding audio speech recognition in noisy scenarios. Multiple camera setups can be beneficial for the visual speech recognition systems in terms of improved performance and robustness. In this paper, we explore this aspect and provide a comprehensive study on combining multiple views for visual speech recognition. The thorough analysis covers fusion of all possible view angle combinations both at feature level and decision level. The employed visual speech recognition system in this study extracts features through a PCA-based convolutional neural network, followed by an LSTM network. Finally, these features are processed in a tandem system, being fed into a GMM-HMM scheme. The decision fusion acts after this point by combining the Viterbi path log-likelihoods. The results show that the complementary information contained in recordings from different view angles improves the results significantly. For example, the sentence correctness on the test set is increased from 76% for the highest performing single view ($30^\circ$) to up to 83% when combining this view with the frontal and $60^\circ$ view angles.
Code (0)
등록된 구현이 없습니다.
Tasks
Sentencespeech-recognitionSpeech RecognitionVisual Speech RecognitionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
End-to-End Audiovisual Fusion with LSTMs
Several end-to-end deep learning approaches have been recently presented which simultaneously extract visual features from the input images and perform visual speech classification. However, research on jointly extractin…
ClassificationGeneral Classificationspeech-recognitionSpeech RecognitionMulti-modal Speech Transformer Decoders: When Do Multiple Modalities Improve Accuracy?
Decoder-only discrete-token language models have recently achieved significant success in automatic speech recognition. However, systematic analyses of how different modalities impact performance in specific scenarios re…
Automatic Speech RecognitionDecoderspeech-recognitionSpeech RecognitionCR-CTC: Consistency regularization on CTC for improved speech recognition
Connectionist Temporal Classification (CTC) is a widely used method for automatic speech recognition (ASR), renowned for its simplicity and computational efficiency. However, it often falls short in recognition performan…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Computational EfficiencyDecoder+2Emotion Recognition System from Speech and Visual Information based on Convolutional Neural Networks
Emotion recognition has become an important field of research in the human-computer interactions domain. The latest advancements in the field show that combining visual with audio information lead to better results if co…
Emotion RecognitionMulti-view Frequency LSTM: An Efficient Frontend for Automatic Speech Recognition
Acoustic models in real-time speech recognition systems typically stack multiple unidirectional LSTM layers to process the acoustic frames over time. Performance improvements over vanilla LSTM architectures have been rep…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)speech-recognitionSpeech Recognition