Earnings-22: A Practical Benchmark for Accents in the Wild
Modern automatic speech recognition (ASR) systems have achieved superhuman Word Error Rate (WER) on many common corpora despite lacking adequate performance on speech in the wild. Beyond that, there is a lack of real-world, accented corpora to properly benchmark academic and commercial models. To ensure this type of speech is represented in ASR benchmarking, we present Earnings-22, a 125 file, 119 hour corpus of English-language earnings calls gathered from global companies. We run a comparison across 4 commercial models showing the variation in performance when taking country of origin into consideration. Looking at hypothesis transcriptions, we explore errors common to all ASR systems tested. By examining Individual Word Error Rate (IWER), we find that key speech features impact model performance more for certain accents than others. Earnings-22 provides a free-to-use benchmark of real-world, accented audio to bridge academic and industrial research.
Code (1)
Tasks
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Benchmarkingspeech-recognitionSpeech RecognitionSimilar Papers 제목 키워드 기반
Earnings-21: A Practical Benchmark for ASR in the Wild
Commonly used speech corpora inadequately challenge academic and commercial ASR systems. In particular, speech corpora lack metadata needed for detailed analysis and WER measurement. In response, we present Earnings-21, …
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)NERAfriVox-v2: A Domain-Verticalized Benchmark for In-the-Wild African Speech Recognition
Recent large language models (LLMs) show strong speech recognition and translation capabilities for high-resource languages. However, African languages remain dramatically underrepresented in benchmarks, limiting their p…
Speech RecognitionContextual Earnings-22: A Speech Recognition Benchmark with Custom Vocabulary in the Wild
The accuracy frontier of speech-to-text systems has plateaued on academic benchmarks.1 In contrast, industrial benchmarks and adoption in high-stakes domains suggest otherwise. We hypothesize that the primary difference …
Speech RecognitionBenchmarking_Fast_Domain_Adaptation_for_Unsupervised_Speech_Units
Representation learning has attracted great atten- tion and managed to reach good performances as a pretraining method for downstream tasks or as a first step towards unsu- pervised speech modeling. Yet, little is known …
Representation LearningAccentFold: A Journey through African Accents for Zero-Shot ASR Adaptation to Target Accents
Despite advancements in speech recognition, accented speech remains challenging. While previous approaches have focused on modeling techniques or creating accented speech datasets, gathering sufficient data for the multi…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Diversityspeech-recognition+1