SD-QA: Spoken Dialectal Question Answering for the Real World
Question answering (QA) systems are now available through numerous commercial applications for a wide variety of domains, serving millions of users that interact with them via speech interfaces. However, current benchmarks in QA research do not account for the errors that speech recognition models might introduce, nor do they consider the language variations (dialects) of the users. To address this gap, we augment an existing QA dataset to construct a multi-dialect, spoken QA benchmark on five languages (Arabic, Bengali, English, Kiswahili, Korean) with more than 68k audio prompts in 24 dialects from 255 speakers. We provide baseline results showcasing the real-world performance of QA systems and analyze the effect of language variety and other sensitive speaker attributes on downstream performance. Last, we study the fairness of the ASR and QA models with respect to the underlying user populations. The dataset, model outputs, and code for reproducing all our experiments are available: https://github.com/ffaisal93/SD-QA.
Code (1)
Tasks
FairnessQuestion Answeringspeech-recognitionSpeech RecognitionSimilar Papers 제목 키워드 기반
HeySQuAD: A Spoken Question Answering Dataset
Spoken question answering (SQA) systems are critical for digital assistants and other real-world use cases, but evaluating their performance is a challenge due to the importance of human-spoken questions. This study pres…
Question AnsweringOASIS: A Multilingual and Multimodal Dataset for Culturally Grounded Spoken Visual QA
Large-scale multimodal models achieve strong results on tasks like Visual Question Answering (VQA), but they are often limited when queries require cultural and visual information, everyday knowledge, particularly in low…
Visual Question AnsweringObject RecognitionDUAL: Textless Spoken Question Answering with Speech Discrete Unit Adaptive Learning
Spoken Question Answering (SQA) has gained research attention and made remarkable progress in recent years. However, existing SQA methods rely on Automatic Speech Recognition (ASR) transcriptions, which is time and cost-…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Question Answeringspeech-recognition+1Worldly Wise (WoW) - Cross-Lingual Knowledge Fusion for Fact-based Visual Spoken-Question Answering
Although Question-Answering has long been of research interest, its accessibility to users through a speech interface and its support to multiple languages have not been addressed in prior studies. Towards these ends, we…
Knowledge GraphsQuestion AnsweringSpeech-to-TextSpoken Language Understanding+1DUAL: Discrete Spoken Unit Adaptive Learning for Textless Spoken Question Answering
Spoken Question Answering (SQA) is to find the answer from a spoken document given a question, which is crucial for personal assistants when replying to the queries from the users. Existing SQA methods all rely on Automa…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Question Answeringspeech-recognition+1