SPES: Spectrogram Perturbation for Explainable Speech-to-Text Generation
Spurred by the demand for interpretable models, research on eXplainable AI for language technologies has experienced significant growth, with feature attribution methods emerging as a cornerstone of this progress. While prior work in NLP explored such methods for classification tasks and textual applications, explainability intersecting generation and speech is lagging, with existing techniques failing to account for the autoregressive nature of state-of-the-art models and to provide fine-grained, phonetically meaningful explanations. We address this gap by introducing Spectrogram Perturbation for Explainable Speech-to-text Generation (SPES), a feature attribution technique applicable to sequence generation tasks with autoregressive models. SPES provides explanations for each predicted token based on both the input spectrogram and the previously generated tokens. Extensive evaluation on speech recognition and translation demonstrates that SPES generates explanations that are faithful and plausible to humans.
Code (1)
Tasks
speech-recognitionSpeech RecognitionSpeech-to-TextText GenerationSimilar Papers 제목 키워드 기반
Phoneme Discretized Saliency Maps for Explainable Detection of AI-Generated Voice
In this paper, we propose Phoneme Discretized Saliency Maps (PDSM), a discretization algorithm for saliency maps that takes advantage of phoneme boundaries for explainable detection of AI-generated voice. We experimental…
text-to-speechText to SpeechPhaseFool: Phase-oriented Audio Adversarial Examples via Energy Dissipation
Audio adversarial attacks design perturbations onto inputs that lead an automatic speech recognition (ASR) model to predict incorrect outputs. Current audio adversarial attacks optimize perturbations with different const…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Sentencespeech-recognition+1X-DC: Explainable Deep Clustering based on Learnable Spectrogram Templates
Deep neural networks (DNNs) have achieved substantial predictive performance in various speech processing tasks. Particularly, it has been shown that a monaural speech separation task can be successfully solved with a DN…
ClusteringDeep ClusteringSpeech SeparationRadically Old Way of Computing Spectra: Applications in End-to-End ASR
We propose a technique to compute spectrograms using Frequency Domain Linear Prediction (FDLP) that uses all-pole models to fit the squared Hilbert envelope of speech in different frequency sub-bands. The spectrogram of …
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)speech-recognitionSpeech RecognitionThe Unheard Alternative: Contrastive Explanations for Speech-to-Text Models
Contrastive explanations, which indicate why an AI system produced one output (the target) instead of another (the foil), are widely regarded in explainable AI as more informative and interpretable than standard explanat…