Semantic Complexity in End-to-End Spoken Language Understanding
End-to-end spoken language understanding (SLU) models are a class of model architectures that predict semantics directly from speech. Because of their input and output types, we refer to them as speech-to-interpretation (STI) models. Previous works have successfully applied STI models to targeted use cases, such as recognizing home automation commands, however no study has yet addressed how these models generalize to broader use cases. In this work, we analyze the relationship between the performance of STI models and the difficulty of the use case to which they are applied. We introduce empirical measures of dataset semantic complexity to quantify the difficulty of the SLU tasks. We show that near-perfect performance metrics for STI models reported in the literature were obtained with datasets that have low semantic complexity values. We perform experiments where we vary the semantic complexity of a large, proprietary dataset and show that STI model performance correlates with our semantic complexity measures, such that performance increases as complexity values decrease. Our results show that it is important to contextualize an STI model's performance with the complexity values of its training dataset to reveal the scope of its applicability.
Code (0)
등록된 구현이 없습니다.
Tasks
Spoken Language UnderstandingSimilar Papers 제목 키워드 기반
Prompting Whisper for QA-driven Zero-shot End-to-end Spoken Language Understanding
Zero-shot spoken language understanding (SLU) enables systems to comprehend user utterances in new domains without prior exposure to training data. Recent studies often rely on large language models (LLMs), leading to ex…
Cross-corpusDecoderQuestion Answeringslot-filling+2MAC-SLU: Multi-Intent Automotive Cabin Spoken Language Understanding Benchmark
Spoken Language Understanding (SLU), which aims to extract user semantics to execute downstream tasks, is a crucial component of task-oriented dialog systems. Existing SLU datasets generally lack sufficient diversity and…
Spoken Language UnderstandingSpeech RecognitionA Study on the Integration of Pipeline and E2E SLU systems for Spoken Semantic Parsing toward STOP Quality Challenge
Recently there have been efforts to introduce new benchmark tasks for spoken language understanding (SLU), like semantic parsing. In this paper, we describe our proposed spoken semantic parsing system for the quality tra…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Semantic Parsingspeech-recognition+2Fast Intent Classification for Spoken Language Understanding
Spoken Language Understanding (SLU) systems consist of several machine learning components operating together (e.g. intent classification, named entity recognition and resolution). Deep learning models have obtained stat…
ClassificationDecision MakingGeneral Classificationintent-classification+6End-to-End Spoken Language Understanding for Generalized Voice Assistants
End-to-end (E2E) spoken language understanding (SLU) systems predict utterance semantics directly from speech using a single model. Previous work in this area has focused on targeted tasks in fixed domains, where the out…
Spoken Language Understanding