Investigating Decoder-only Large Language Models for Speech-to-text Translation
Large language models (LLMs), known for their exceptional reasoning capabilities, generalizability, and fluency across diverse domains, present a promising avenue for enhancing speech-related tasks. In this paper, we focus on integrating decoder-only LLMs to the task of speech-to-text translation (S2TT). We propose a decoder-only architecture that enables the LLM to directly consume the encoded speech representation and generate the text translation. Additionally, we investigate the effects of different parameter-efficient fine-tuning techniques and task formulation. Our model achieves state-of-the-art performance on CoVoST 2 and FLEURS among models trained without proprietary data. We also conduct analyses to validate the design choices of our proposed model and bring insights to the integration of LLMs to S2TT.
Code (0)
등록된 구현이 없습니다.
Tasks
Decoderparameter-efficient fine-tuningSpeech-to-TextSpeech-to-Text TranslationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
On decoder-only architecture for speech-to-text and large language model integration
Large language models (LLMs) have achieved remarkable success in the field of natural language processing, enabling better human-computer interaction using natural language. However, the seamless integration of speech si…
DecoderLanguage ModelingLanguage ModellingLarge Language Model+2Homophone Disambiguation Reveals Patterns of Context Mixing in Speech Transformers
Transformers have become a key architecture in speech processing, but our understanding of how they build up representations of acoustic and linguistic structure is limited. In this study, we address this gap by investig…
Decoderspeech-recognitionSpeech RecognitionMulti-Encoder Learning and Stream Fusion for Transformer-Based End-to-End Automatic Speech Recognition
Stream fusion, also known as system combination, is a common technique in automatic speech recognition for traditional hybrid hidden Markov model approaches, yet mostly unexplored for modern deep neural network end-to-en…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)DecoderLanguage Modeling+3Measuring the Redundancy of Decoder Layers in SpeechLLMs
Speech Large Language Models route speech encoder representations into an LLM decoder that typically accounts for over 90% of total parameters. We study how much of this decoder capacity is actually needed for speech tas…
Using Large Language Model for End-to-End Chinese ASR and NER
Mapping speech tokens to the same feature space as text tokens has become the paradigm for the integration of speech modality into decoder-only large language models (LLMs). An alternative approach is to use an encoder-d…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)DecoderLanguage Modeling+5