Do Prompts Really Prompt? Exploring the Prompt Understanding Capability of Whisper
This research explores how the information of prompts interacts with the high-performing speech recognition model, Whisper. We compare its performances when prompted by prompts with correct information and those corrupted with incorrect information. Our results unexpectedly show that Whisper may not understand the textual prompts in a human-expected way. Additionally, we find that performance improvement is not guaranteed even with stronger adherence to the topic information in textual prompts. It is also noted that English prompts generally outperform Mandarin ones on datasets of both languages, likely due to differences in training data distributions for these languages despite the mismatch with pre-training scenarios. Conversely, we discover that Whisper exhibits awareness of misleading information in language tokens by ignoring incorrect language tokens and focusing on the correct ones. In sum, We raise insightful questions about Whisper's prompt understanding and reveal its counter-intuitive behaviors. We encourage further studies.
Code (1)
Tasks
speech-recognitionSpeech RecognitionSimilar Papers 제목 키워드 기반
Do Prompt-Based Models Really Understand the Meaning of Their Prompts?
Recently, a boom of papers has shown extraordinary progress in zero-shot and few-shot learning with various prompt-based models. Such success can give the impression that prompts help models to learn faster in the same w…
Few-Shot LearningNatural Language InferenceDo Prompt-Based Models Really Understand the Meaning of their Prompts?
Recently, a boom of papers has shown extraordinary progress in zero-shot and few-shot learning with various prompt-based models. It is commonly argued that prompts help models to learn faster in the same way that humans …
Few-Shot LearningNatural Language InferenceIteraTTA: An interface for exploring both text prompts and audio priors in generating music with text-to-audio models
Recent text-to-audio generation techniques have the potential to allow novice users to freely generate music audio. Even if they do not have musical knowledge, such as about chord progressions and instruments, users can …
Audio GenerationMusic GenerationDo We Really Need a Large Number of Visual Prompts?
Due to increasing interest in adapting models on resource-constrained edges, parameter-efficient transfer learning has been widely explored. Among various methods, Visual Prompt Tuning (VPT), prepending learnable prompts…
Transfer LearningVisual Prompt TuningDon't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models
Recent advancements in generative AI have enabled ubiquitous access to large language models (LLMs). Empowered by their exceptional capabilities to understand and generate human-like text, these models are being increasi…