Back Transcription as a Method for Evaluating Robustness of Natural Language Understanding Models to Speech Recognition Errors

2023-10-25 · Marek Kubis, Paweł Skórzewski, Marcin Sowański, Tomasz Ziętkiewicz

In a spoken dialogue system, an NLU model is preceded by a speech recognition system that can deteriorate the performance of natural language understanding. This paper proposes a method for investigating the impact of speech recognition errors on the performance of natural language understanding models. The proposed method combines the back transcription procedure with a fine-grained technique for categorizing the errors that affect the performance of NLU models. The method relies on the usage of synthesized speech for NLU evaluation. We show that the use of synthesized speech in place of audio recording does not change the outcomes of the presented technique in a significant way.

📄 PDF Abstract BibTeX arXiv:2310.16609

Code (2)

marekkubis/bteval 공식 구현

QData/TextAttack/blob/master/textattack/transformations/sentence_transformations/back_transcription.py

Tasks

en-US domain classificationen-US Intent Classificationen-US Slot FillingNatural Language Understanding speech-recognition Speech Recognition Spoken Language Understanding Text-To-Speech Synthesis

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Position-Wise Feed-Forward Layer 설명 없음

Residual Connection 설명 없음

Multi-Head Attention 설명 없음

Attention 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Music-robust Automatic Lyrics Transcription of Polyphonic Music

2022-04-07 · Xiaoxue Gao, Chitralekha Gupta, Haizhou Li

Lyrics transcription of polyphonic music is challenging because singing vocals are corrupted by the background music. To improve the robustness of lyrics transcription to the background music, we propose a strategy of co…

Automatic Lyrics Transcription Language Modeling Language Modelling

Correcting Automated and Manual Speech Transcription Errors using Warped Language Models

2021-03-26 · Mahdi Namazifar, John Malik, Li Erran Li, Gokhan Tur 외

Masked language models have revolutionized natural language processing systems in the past few years. A recently introduced generalization of masked language models called warped language models are trained to be more ro…

Language Modeling Language Modelling

K-Function: Joint Pronunciation Transcription and Feedback for Evaluating Kids Language Function

2025-07-03 · Shuhe Li, Chenxu Guo, Jiachen Lian, Cheol Jun Cho 외 arxiv

Evaluating young children's language is challenging for automatic speech recognizers due to high-pitched voices, prolonged sounds, and limited data. We introduce K-Function, a framework that combines accurate sub-word tr…

Towards a General-Purpose Linguistic Annotation Backend

2018-12-13 · Graham Neubig, Patrick Littell, Chian-Yu Chen, Jean Lee 외

Language documentation is inherently a time-intensive process; transcription, glossing, and corpus management consume a significant portion of documentary linguists' work. Advances in natural language processing can help…

Management

Evaluating Fine-Tuned LLM Model For Medical Transcription With Small Low-Resource Languages Validated Dataset

2026-03-25 · Mohammed Nowshad Ruhani Chowdhury, Mohammed Nowaz Rabbani Chowdhury, Sakari Lukkarinen arxiv

Clinical documentation is a critical factor for patient safety, diagnosis, and continuity of care. The administrative burden of EHRs is a significant factor in physician burnout. This is a critical issue for low-resource…

Semantic Similarity