How Do Instructions Shape Speech? Cross-Attention Attribution for Style-Captioned Text-to-Speech
Style-captioned text-to-speech systems use natural language to control voice characteristics, but how individual words influence acoustic output remains unclear. Understanding this is critical for diagnosing failure modes and improving controllability in expressive TTS. We propose cross-attention attribution for speech diffusion models, adapting the DAAM framework to the speech domain for the first time, and apply it to CapSpeech-TTS. Our method extracts per-token heatmaps across 25 layers and 24 ODE steps. We analyze 3,600 (style caption, text transcript) combinations comprising 120 style captions conditioning the generation of 30 text transcripts each, revealing how caption tokens shape waveforms. Results show: (1) style tokens have lower temporal variance than content/function tokens, confirming global conditioning; (2) style attention correlates with F0 and energy; (3) style conditioning peaks in early steps and deep layers; (4) attention entropy reaches its minimum at layer 17, co-occurring with the style importance peak, indicating maximal network selectivity at the most style-critical stage. This is the first study of how natural language influences cross-attention in speech diffusion models
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Explainable AI: Context-Aware Layer-Wise Integrated Gradients for Explaining Transformer Models
Transformer models achieve state-of-the-art performance across domains and tasks, yet their deeply layered representations make their predictions difficult to interpret. Existing explainability methods rely on final-laye…
Document ClassificationHate Speech DetectionImage ClassificationSentiment AnalysisThe Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentation
Large language models are able to exploit in-context learning to access external knowledge beyond their training data through retrieval-augmentation. While promising, its inner workings remain unclear. In this work, we s…
Answer GenerationIn-Context LearningQuestion AnsweringRetrievalBenchmarking Post-Hoc Interpretability Approaches for Transformer-based Misogyny Detection
Transformer-based Natural Language Processing models have become the standard for hate speech detection. However, the unconscious use of these techniques for such a critical task comes with negative consequences. Various…
BenchmarkingHate Speech DetectionCross-Attention is Half Explanation in Speech-to-Text Models
Cross-attention is a core mechanism in encoder-decoder architectures, widespread in many fields, including speech-to-text (S2T) processing. Its scores have been repurposed for various downstream applications--such as tim…
Listening with Attention: Entropy-Guided Explainability for Transformer-Based Audio Models
Transformer-based automatic speech recognition (ASR) models such as Whisper are highly accurate, but their predictions remain difficult to interpret. Existing explainable AI (XAI) methods often lack faithfulness and prec…
Speech Recognition