WAND: Windowed Attention and Knowledge Distillation for Efficient Autoregressive Text-to-Speech Models
Recent decoder-only autoregressive text-to-speech (AR-TTS) models produce high-fidelity speech, but their memory and compute costs scale quadratically with sequence length due to full self-attention. In this paper, we propose WAND, Windowed Attention and Knowledge Distillation, a framework that adapts pretrained AR-TTS models to operate with constant computational and memory complexity. WAND separates the attention mechanism into two: persistent global attention over conditioning tokens and local sliding-window attention over generated tokens. To stabilize fine-tuning, we employ a curriculum learning strategy that progressively tightens the attention window. We further utilize knowledge distillation from a full-attention teacher to recover high-fidelity synthesis quality with high data efficiency. Evaluated on three modern AR-TTS models, WAND preserves the original quality while achieving up to 66.2% KV cache memory reduction and length-invariant, near-constant per-step latency.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationSimilar Papers 제목 키워드 기반
AfroXLMR-Comet: Multilingual Knowledge Distillation with Attention Matching for Low-Resource languages
Language model compression through knowledge distillation has emerged as a promising approach for deploying large language models in resource-constrained environments. However, existing methods often struggle to maintain…
Knowledge DistillationLanguage ModelingLanguage ModellingModel CompressionConformer with dual-mode chunked attention for joint online and offline ASR
In this paper, we present an in-depth study on online attention mechanisms and distillation techniques for dual-mode (i.e., joint online and offline) ASR using the Conformer Transducer. In the dual-mode Conformer Transdu…
Knowledge DistillationSWARD: Stochastic Window-Attention-Based Relational Distillation for Cross-Architectural Semantic Segmentation
Large-scale vision foundation models have driven substantial gains on dense prediction tasks such as semantic segmentation, but their size makes deployment impractical in resource-constrained settings, motivating knowled…
Medical Image SegmentationKnowledge DistillationSemantic SegmentationScene ParsingUnderstanding Knowledge Distillation in Non-autoregressive Machine Translation
Non-autoregressive machine translation (NAT) systems predict a sequence of output tokens in parallel, achieving substantial improvements in generation speed compared to autoregressive models. Existing NAT models usually …
Knowledge DistillationMachine TranslationTranslationData-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation
We study the transformation of autoregressive models (ARLMs) into diffusion language models (DLMs). Rather than pretraining from scratch, prior work replaces the causal attention in ARLMs with bidirectional attention and…