Semantic Segmentation with Bidirectional Language Models Improves Long-form ASR
We propose a method of segmenting long-form speech by separating semantically complete sentences within the utterance. This prevents the ASR decoder from needlessly processing faraway context while also preventing it from missing relevant context within the current sentence. Semantically complete sentence boundaries are typically demarcated by punctuation in written text; but unfortunately, spoken real-world utterances rarely contain punctuation. We address this limitation by distilling punctuation knowledge from a bidirectional teacher language model (LM) trained on written, punctuated text. We compare our segmenter, which is distilled from the LM teacher, against a segmenter distilled from a acoustic-pause-based teacher used in other works, on a streaming ASR pipeline. The pipeline with our segmenter achieves a 3.2% relative WER gain along with a 60 ms median end-of-segment latency reduction on a YouTube captioning task.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderFormLanguage ModelingLanguage ModellingSemantic SegmentationSentenceSimilar Papers 제목 키워드 기반
MSD-KMamba: Bidirectional Spatial-Aware Multi-Modal 3D Brain Segmentation via Multi-scale Self-Distilled Fusion Strategy
Numerous CNN-Transformer hybrid models rely on high-complexity global attention mechanisms to capture long-range dependencies, which introduces non-linear computational complexity and leads to significant resource consum…
Computational EfficiencyKnowledge DistillationBrain SegmentationImage SegmentationFlowSeg: Dynamic Semantic Guidance for LLM-Conditioned Segmentation
LLM-conditioned segmentation has recently advanced rapidly by coupling large language models with iterative mask generation frameworks. However, we identify a persistent failure mode in current propose-then-select pipeli…
Referring Expression SegmentationBiCLIP: Bidirectional and Consistent Language-Image Processing for Robust Medical Image Segmentation
Medical image segmentation is a cornerstone of computer-assisted diagnosis and treatment planning. While recent multimodal vision-language models have shown promise in enhancing semantic understanding through textual des…
Medical Image SegmentationContext Patch Fusion With Class Token Enhancement for Weakly Supervised Semantic Segmentation
Weakly Supervised Semantic Segmentation (WSSS), which relies only on image-level labels, has attracted significant attention for its cost-effectiveness and scalability. Existing methods mainly enhance inter-class distinc…
Semantic SegmentationData AugmentationBiCoR-Seg: Bidirectional Co-Refinement Framework for High-Resolution Remote Sensing Image Segmentation
High-resolution remote sensing image semantic segmentation (HRSS) is a fundamental yet critical task in the field of Earth observation. However, it has long faced the challenges of high inter-class similarity and large i…
Semantic SegmentationImage Segmentation