LAST: Language Model Aware Speech Tokenization
Speech tokenization serves as the foundation of speech language model (LM), enabling them to perform various tasks such as spoken language modeling, text-to-speech, speech-to-text, etc. Most speech tokenizers are trained independently of the LM training process, relying on separate acoustic models and quantization methods. Following such an approach may create a mismatch between the tokenization process and its usage afterward. In this study, we propose a novel approach to training a speech tokenizer by leveraging objectives from pre-trained textual LMs. We advocate for the integration of this objective into the process of learning discrete speech representations. Our aim is to transform features from a pre-trained speech model into a new feature space that enables better clustering for speech LMs. We empirically investigate the impact of various model design choices, including speech vocabulary size and text LM size. Our results demonstrate the proposed tokenization method outperforms the evaluated baselines considering both spoken language modeling and speech-to-text. More importantly, unlike prior work, the proposed method allows the utilization of a single pre-trained LM for processing both speech and text inputs, setting it apart from conventional tokenization approaches.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingmodelQuantizationSpeech TokenizationSpeech-to-Texttext-to-speechText to SpeechSimilar Papers 제목 키워드 기반
Sylber: Syllabic Embedding Representation of Speech from Raw Audio
Syllables are compositional units of spoken language that efficiently structure human speech perception and production. However, current neural speech representations lack such structure, resulting in dense token sequenc…
Language ModelingLanguage ModellingSelf-Supervised LearningSpeech TokenizationExploring the Effect of Segmentation and Vocabulary Size on Speech Tokenization for Speech Language Models
The purpose of speech tokenization is to transform a speech signal into a sequence of discrete representations, serving as the foundation for speech language models (SLMs). While speech tokenization has many options, the…
Speech TokenizationSpoken Language UnderstandingStreamAlign: Streaming Text-Aligned Speech Tokenization
Text-aligned speech tokenization methods have emerged to better align speech tokens with LLM token spaces, enabling more effective utilization of pretrained LLMs. However, they rely on offline automatic speech recognitio…
Speech RecognitionRepCodec: A Speech Representation Codec for Speech Tokenization
With recent rapid growth of large language models (LLMs), discrete speech tokenization has played an important role for injecting speech into LLMs. However, this discretization gives rise to a loss of information, conseq…
Language ModelingLanguage ModellingQuantizationSpeech TokenizationTASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling
Recent efforts target spoken language models (SLMs) that not only listen but also speak for more natural human-LLM interaction. Joint speech-text modeling is a promising direction to achieve this. However, the effectiven…
Language ModelingLanguage Modellingparameter-efficient fine-tuningSpeech Tokenization