Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Information
Quantization in SSL speech models (e.g., HuBERT) improves compression and performance in tasks like language modeling, resynthesis, and text-to-speech but often discards prosodic and paralinguistic information (e.g., emotion, prominence). While increasing codebook size mitigates some loss, it inefficiently raises bitrates. We propose Segmentation-Variant Codebooks (SVCs), which quantize speech at distinct linguistic units (frame, phone, word, utterance), factorizing it into multiple streams of segment-specific discrete features. Our results show that SVCs are significantly more effective at preserving prosodic and paralinguistic information across probing tasks. Additionally, we find that pooling before rather than after discretization better retains segment-level information. Resynthesis experiments further confirm improved style realization and slightly improved quality while preserving intelligibility.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingQuantizationResynthesistext-to-speechText to SpeechSimilar Papers 제목 키워드 기반
Improving Speech Emotion Recognition with Unsupervised Speaking Style Transfer
Humans can effortlessly modify various prosodic attributes, such as the placement of stress and the intensity of sentiment, to convey a specific emotion while maintaining consistent linguistic content. Motivated by this …
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Data AugmentationDecoder+6A Variational Prosody Model for the decomposition and synthesis of speech prosody
The quest for comprehensive generative models of intonation that link linguistic and paralinguistic functions to prosodic forms has been a longstanding challenge of speech communication research. More traditional intonat…
Speech SynthesisNPU-NTU System for Voice Privacy 2024 Challenge
Speaker anonymization is an effective privacy protection solution that conceals the speaker's identity while preserving the linguistic content and paralinguistic information of the original speech. To establish a fair be…
DisentanglementSpeaker anonymizationThe Impact of Prosodic Segmentation on Speech Synthesis of Spontaneous Speech
Spontaneous speech presents several challenges for speech synthesis, particularly in capturing the natural flow of conversation, including turn-taking, pauses, and disfluencies. Although speech synthesis systems have mad…
Speech SynthesisA Variational Framework for Improving Naturalness in Generative Spoken Language Models
The success of large language models in text processing has inspired their adaptation to speech modeling. However, since speech is continuous and complex, it is often discretized for autoregressive modeling. Speech token…