Enhancing Large Language Model-based Speech Recognition by Contextualization for Rare and Ambiguous Words
We develop a large language model (LLM) based automatic speech recognition (ASR) system that can be contextualized by providing keywords as prior information in text prompts. We adopt decoder-only architecture and use our in-house LLM, PLaMo-100B, pre-trained from scratch using datasets dominated by Japanese and English texts as the decoder. We adopt a pre-trained Whisper encoder as an audio encoder, and the audio embeddings from the audio encoder are projected to the text embedding space by an adapter layer and concatenated with text embeddings converted from text prompts to form inputs to the decoder. By providing keywords as prior information in the text prompts, we can contextualize our LLM-based ASR system without modifying the model architecture to transcribe ambiguous words in the input audio accurately. Experimental results demonstrate that providing keywords to the decoder can significantly improve the recognition performance of rare and ambiguous words.
Code (0)
등록된 구현이 없습니다.
Tasks
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)DecoderLanguage ModelingLanguage ModellingLarge Language Modelspeech-recognitionSpeech RecognitionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusion
How to leverage dynamic contextual information in end-to-end speech recognition has remained an active research area. Previous solutions to this problem were either designed for specialized use cases that did not general…
Language ModelingLanguage Modellingspeech-recognitionSpeech RecognitionContextualization of ASR with LLM using phonetic retrieval-based augmentation
Large language models (LLMs) have shown superb capability of modeling multimodal signals including audio and text, allowing the model to generate spoken or textual response given a speech input. However, it remains a cha…
Retrievalspeech-recognitionSpeech RecognitionEnd-to-End Speech Recognition Contextualization with Large Language Models
In recent years, Large Language Models (LLMs) have garnered significant attention from the research community due to their exceptional performance and generalization capabilities. In this paper, we introduce a novel meth…
DecoderLanguage ModelingLanguage Modellingspeech-recognition+1DYNAC: Dynamic Vocabulary based Non-Autoregressive Contextualization for Speech Recognition
Contextual biasing (CB) improves automatic speech recognition for rare and unseen phrases. Recent studies have introduced dynamic vocabulary, which represents context phrases as expandable tokens in autoregressive (AR) m…
Automatic Speech Recognitionspeech-recognitionSpeech RecognitionASR Adaptation for E-commerce Chatbots using Cross-Utterance Context and Multi-Task Language Modeling
Automatic Speech Recognition (ASR) robustness toward slot entities are critical in e-commerce voice assistants that involve monetary transactions and purchases. Along with effective domain adaptation, it is intuitive tha…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Domain AdaptationLanguage Modeling+3