Papers Lipreading
“Lipreading” 태그가 달린 논문 103편 · 필터 해제
Learning Speaker-Invariant Visual Features for Lipreading
Lipreading is a challenging cross-modal task that aims to convert visual lip movements into spoken text. Existing lipreading methods often extract visual features that include speaker-specific lip attributes (e.g., shape…
DisentanglementLipreadingSpeaker RecognitionUniCUE: Unified Recognition and Generation Framework for Chinese Cued Speech Video-to-Speech Generation
Cued Speech (CS) enhances lipreading through hand coding, providing precise speech perception support for the hearing-impaired. CS Video-to-Speech generation (CSV2S) task aims to convert the CS visual expressions (CS vid…
cross-modal alignmentLipreadingtext-to-speechText to SpeechOXSeg: Multidimensional attention UNet-based lip segmentation using semi-supervised lip contours
Lip segmentation plays a crucial role in various domains, such as lip synchronization, lipreading, and diagnostics. However, the effectiveness of supervised lip segmentation is constrained by the availability of lip cont…
Generative Adversarial NetworkLipreadingSegmentationAudio-Visual Representation Learning via Knowledge Distillation from Speech Foundation Models
Audio-visual representation learning is crucial for advancing multimodal speech processing tasks, such as lipreading and audio-visual speech recognition. Recently, speech foundation models (SFMs) have shown remarkable ge…
Audio-Visual Speech RecognitionAutomatic Speech RecognitionAutomatic Speech Recognition (ASR)Knowledge Distillation+5Target Speaker Lipreading by Audio-Visual Self-Distillation Pretraining and Speaker Adaptation
Lipreading is an important technique for facilitating human-computer interaction in noisy environments. Our previously developed self-supervised learning method, AV2vec, which leverages multimodal self-distillation, has …
Cross-Lingual TransferLipreadingSelf-Supervised LearningTransfer LearningEvaluation of End-to-End Continuous Spanish Lipreading in Different Data Conditions
Visual speech recognition remains an open research problem where different challenges must be considered by dispensing with the auditory sense, such as visual ambiguities, the inter-personal variability among speakers, a…
Lipreadingspeech-recognitionSpeech RecognitionVisual Speech RecognitionUnified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs
Research in auditory, visual, and audiovisual speech recognition (ASR, VSR, and AVSR, respectively) has traditionally been conducted independently. Even recent self-supervised studies addressing two or all three tasks si…
Lipreadingspeech-recognitionSpeech RecognitionRAL:Redundancy-Aware Lipreading Model Based on Differential Learning with Symmetric Views
Lip reading involves interpreting a speaker's speech by analyzing sequences of lip movements. Currently, most models regard the left and right halves of the lips as a symmetrical whole, lacking a thorough investigation o…
LipreadingLip ReadingSyncVSR: Data-Efficient Visual Speech Recognition with End-to-End Crossmodal Audio Token Synchronization
Visual Speech Recognition (VSR) stands at the intersection of computer vision and speech recognition, aiming to interpret spoken content from visual cues. A prominent challenge in VSR is the presence of homophenes-visual…
Landmark-based LipreadingLipreadingspeech-recognitionSpeech Recognition+1Watch Your Mouth: Silent Speech Recognition with Depth Sensing
Silent speech recognition is a promising technology that decodes human speech without requiring audio signals, enabling private human-computer interactions. In this paper, we propose Watch Your Mouth, a novel method that…
Deep LearningLipreadingSilent Speech Recognitionspeech-recognition+2Audio-Visual Speech Recognition based on Regulated Transformer and Spatio-Temporal Fusion Strategy for Driver Assistive Systems
This article presents a research methodology for audio-visual speech recognition (AVSR) in driver assistive systems. These systems necessitate ongoing interaction with drivers while driving through voice control for safe…
Audio-Visual Speech RecognitionLipreadingLip Readingspeech-recognition+2Enhancing Lip Reading with Multi-Scale Video and Multi-Encoder
Automatic lip-reading (ALR) aims to automatically transcribe spoken content from a speaker's silent lip motion captured in video. Current mainstream lip-reading approaches only use a single visual encoder to model input …
LipreadingLip ReadingTask 2Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processing
In visual speech processing, context modeling capability is one of the most important requirements due to the ambiguous nature of lip movements. For example, homophenes, words that share identical lip movements but produ…
LipreadingLip Readingspeech-recognitionSpeech Recognition+2Cross-Attention Fusion of Visual and Geometric Features for Large Vocabulary Arabic Lipreading
Lipreading involves using visual data to recognize spoken words by analyzing the movements of the lips and surrounding area. It is a hot research topic with many potential applications, such as human-machine interaction …
LipreadingLip Readingspeech-recognitionSpeech RecognitionES3: Evolving Self-Supervised Learning of Robust Audio-Visual Speech Representations
We propose a novel strategy ES3 for self-supervised learning of robust audio-visual speech representations from unlabeled talking face videos. While many recent approaches for this task primarily rely on guiding the …
Audio-Visual Speech RecognitionLipreadingSelf-Supervised LearningSpeech RecognitionAnalysis of Visual Features for Continuous Lipreading in Spanish
During a conversation, our brain is responsible for combining information obtained from multiple senses in order to improve our ability to understand the message we are perceiving. Different studies have shown the import…
Lipreadingspeech-recognitionSpeech RecognitionVisual Speech RecognitionInvestigating the dynamics of hand and lips in French Cued Speech using attention mechanisms and CTC-based decoding
Hard of hearing or profoundly deaf people make use of cued speech (CS) as a communication tool to understand spoken language. By delivering cues that are relevant to the phonetic information, CS offers a way to enhance l…
LipreadingAudio-Visual Speech Enhancement with Score-Based Generative Models
This paper introduces an audio-visual speech enhancement system that leverages score-based generative models, also known as diffusion models, conditioned on visual information. In particular, we exploit audio-visual embe…
Automatic Speech RecognitionLipreadingSpeech Enhancementspeech-recognition+1Word-level Persian Lipreading Dataset
Lip-reading has made impressive progress in recent years, driven by advances in deep learning. Nonetheless, the prerequisite such advances is a suitable dataset. This paper provides a new in-the-wild dataset for Persian …
LipreadingLip ReadingAuto-AVSR: Audio-Visual Speech Recognition with Automatic Labels
Audio-visual speech recognition has received a lot of attention due to its robustness against acoustic noise. Recently, the performance of automatic, visual, and audio-visual speech recognition (ASR, VSR, and AV-ASR, res…
Audio-Visual Speech RecognitionAutomatic Speech RecognitionAutomatic Speech Recognition (ASR)Lipreading+3