Low Frame-rate Speech Codec: a Codec Designed for Fast High-quality Speech LLM Training and Inference
Large language models (LLMs) have significantly advanced audio processing through audio codecs that convert audio into discrete tokens, enabling the application of language modeling techniques to audio data. However, audio codecs often operate at high frame rates, resulting in slow training and inference, especially for autoregressive models. To address this challenge, we present the Low Frame-rate Speech Codec (LFSC): a neural audio codec that leverages finite scalar quantization and adversarial training with large speech language models to achieve high-quality audio compression with a 1.89 kbps bitrate and 21.5 frames per second. We demonstrate that our novel codec can make the inference of LLM-based text-to-speech models around three times faster while improving intelligibility and producing quality comparable to previous models.
Code (0)
등록된 구현이 없습니다.
Tasks
Audio CompressionLanguage ModelingLanguage ModellingQuantizationtext-to-speechText to SpeechSimilar Papers 제목 키워드 기반
HH-Codec: High Compression High-fidelity Discrete Neural Codec for Spoken Language Modeling
Discrete speech tokenization is a fundamental component in speech codecs. However, in large-scale speech-to-speech systems, the complexity of parallel streams from multiple quantizers and the computational cost of high-t…
TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling
Speech tokenizers serve as foundational components for speech language models, yet current designs exhibit several limitations, including: 1) dependence on multi-layer residual vector quantization structures or high fram…
Codec-SUPERB @ SLT 2024: A lightweight benchmark for neural audio codec models
Neural audio codec models are becoming increasingly important as they serve as tokenizers for audio, enabling efficient transmission or facilitating speech language modeling. The ideal neural audio codec should maintain …
Language ModelingLanguage ModellingFunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec
This paper presents FunCodec, a fundamental neural speech codec toolkit, which is an extension of the open-source speech processing toolkit FunASR. FunCodec provides reproducible training recipes and inference scripts fo…
Automatic Speech Recognitionspeech-recognitionSpeech RecognitionSpeech Synthesis+3LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec
Although discrete speech tokens have exhibited strong potential for language model-based speech generation, their high bitrates and redundant timbre information restrict the development of such models. In this work, we p…
DisentanglementLanguage ModelingLanguage ModellingQuantization+1