Papers Audio-Visual Speech Recognition
“Audio-Visual Speech Recognition” 태그가 달린 논문 118편 · 필터 해제
Candor-LR: A Dyadic Conversational Dataset for Audio-Visual Speech Recognition
Current audio-visual speech recognition (AVSR) benchmarks, like LRS3, rely heavily on clean, scripted and rehearsed speech. They fail to reflect the complexity of natural conversation, which involves overlapping speech, …
Audio-Visual Speech RecognitionVSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness
We introduce VSRo-200, the first large-scale dataset for visual speech recognition (lip reading) in Romanian, comprising 200 hours of real-world podcast videos. All samples are annotated with pseudo-labels generated by a…
Audio-Visual Speech RecognitionDomain GeneralizationLip ReadingVIB-AVSR: Variational Information Bottleneck for Noise-Robust LLM-Based Audio-Visual Speech Recognition
Audio-Visual Speech Recognition takes two input modalities, acoustic and visual streams, where visual information from lip movements aids recognition when audio is noisy. Recently, LLM-based AVSR models have emerged as a…
Audio-Visual Speech RecognitionWhen AVSR Meets Video Conferencing: Dataset, Degradation, and the Hidden Mechanism Behind Performance Collapse
Audio-Visual Speech Recognition (AVSR) has achieved remarkable progress in offline conditions, yet its robustness in real-world video conferencing (VC) remains largely unexplored. This paper presents the first systematic…
Audio-Visual Speech RecognitionSpeech EnhancementDr. SHAP-AV: Decoding Relative Modality Contributions via Shapley Attribution in Audio-Visual Speech Recognition
Audio-Visual Speech Recognition (AVSR) leverages both acoustic and visual information for robust recognition under noise. However, how models balance these modalities remains unclear. We present Dr. SHAP-AV, a framework …
Audio-Visual Speech RecognitionOCR-Enhanced Multimodal ASR Can Read While Listening
Visual information, such as subtitles in a movie, often helps automatic speech recognition. In this paper, we propose Donut-Whisper, an audio-visual ASR model with dual encoder to leverage visual information to improve s…
Audio-Visual Speech RecognitionKnowledge DistillationPurification Before Fusion: Toward Mask-Free Speech Enhancement for Robust Audio-Visual Speech Recognition
Audio-visual speech recognition (AVSR) typically improves recognition accuracy in noisy environments by integrating noise-immune visual cues with audio signals. Nevertheless, high-noise audio inputs are prone to introduc…
Audio-Visual Speech RecognitionSpeech EnhancementScalable Frameworks for Real-World Audio-Visual Speech Recognition
The practical deployment of Audio-Visual Speech Recognition (AVSR) systems is fundamentally challenged by significant performance degradation in real-world environments, characterized by unpredictable acoustic noise and …
Audio-Visual Speech RecognitionOmni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Models
Large language models (LLMs) have recently achieved impressive results in speech recognition across multiple modalities, including Auditory Speech Recognition (ASR), Visual Speech Recognition (VSR), and Audio-Visual Spee…
Audio-Visual Speech RecognitionRepresentation LearningMitigating Attention Sinks and Massive Activations in Audio-Visual Speech Recognition with LLMs
Large language models (LLMs) have recently advanced auditory speech recognition (ASR), visual speech recognition (VSR), and audio-visual speech recognition (AVSR). However, understanding of their internal dynamics under …
Audio-Visual Speech RecognitionTwo Heads Are Better Than One: Audio-Visual Speech Error Correction with Dual Hypotheses
This paper introduces a new paradigm for generative error correction (GER) framework in audio-visual speech recognition (AVSR) that reasons over modality-specific evidences directly in the language space. Our framework, …
Audio-Visual Speech RecognitionMoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition
Large language models (LLMs) have recently shown strong potential in audio-visual speech recognition (AVSR), but their high computational demands and sensitivity to token granularity limit their practicality in resource-…
Audio-Visual Speech RecognitionRepresentation LearningAISHELL6-whisper: A Chinese Mandarin Audio-visual Whisper Speech Dataset with Speech Recognition Baselines
Whisper speech recognition is crucial not only for ensuring privacy in sensitive communications but also for providing a critical communication bridge for patients under vocal restraint and enabling discrete interaction …
Audio-Visual Speech RecognitionReal-Time System for Audio-Visual Target Speech Enhancement
We present a live demonstration for RAVEN, a real-time audio-visual speech enhancement system designed to run entirely on a CPU. In single-channel, audio-only settings, speech enhancement is traditionally approached as t…
Audio-Visual Speech RecognitionSpeech EnhancementTowards Inclusive Communication: A Unified Framework for Generating Spoken Language from Sign, Lip, and Audio
Audio is the primary modality for human communication and has driven the success of Automatic Speech Recognition (ASR) technologies. However, such audio-centric systems inherently exclude individuals who are deaf or hard…
Audio-Visual Speech RecognitionSign Language TranslationText GenerationLip ReadingImproving Noise Robust Audio-Visual Speech Recognition via Router-Gated Cross-Modal Feature Fusion
Robust audio-visual speech recognition (AVSR) in noisy environments remains challenging, as existing systems struggle to estimate audio reliability and dynamically adjust modality reliance. We propose router-gated cross-…
Audio-Visual Speech RecognitionAD-AVSR: Asymmetric Dual-stream Enhancement for Robust Audio-Visual Speech Recognition
Audio-visual speech recognition (AVSR) combines audio-visual modalities to improve speech recognition, especially in noisy environments. However, most existing methods deploy the unidirectional enhancement or symmetric f…
Audio-Visual Speech RecognitionReal-Time Audio-Visual Speech Enhancement Using Pre-trained Visual Representations
Speech enhancement in audio-only settings remains challenging, particularly in the presence of interfering speakers. This paper presents a simple yet effective real-time audio-visual speech enhancement (AVSE) system, RAV…
Audio-Visual Speech RecognitionActive Speaker DetectionSpeech EnhancementViCocktail: Automated Multi-Modal Data Collection for Vietnamese Audio-Visual Speech Recognition
Audio-Visual Speech Recognition (AVSR) has gained significant attention recently due to its robustness against noise, which often challenges conventional speech recognition systems that rely solely on audio features. Des…
Audio-Visual Speech Recognitionspeech-recognitionSpeech RecognitionVisual Speech RecognitionCocktail-Party Audio-Visual Speech Recognition
Audio-Visual Speech Recognition (AVSR) offers a robust solution for speech recognition in challenging environments, such as cocktail-party scenarios, where relying solely on audio proves insufficient. However, current AV…
Audio-Visual Speech Recognitionspeech-recognitionSpeech RecognitionVisual Speech Recognition