Improving Metrics for Speech Translation
We introduce Parallel Paraphrasing ($\text{Para}_\text{both}$), an augmentation method for translation metrics making use of automatic paraphrasing of both the reference and hypothesis. This method counteracts the typically misleading results of speech translation metrics such as WER, CER, and BLEU if only a single reference is available. We introduce two new datasets explicitly created to measure the quality of metrics intended to be applied to Swiss German speech-to-text systems. Based on these datasets, we show that we are able to significantly improve the correlation with human quality perception if our method is applied to commonly used metrics.
Code (0)
등록된 구현이 없습니다.
Tasks
Speech-to-TextTranslationSimilar Papers 제목 키워드 기반
Assessing Evaluation Metrics for Speech-to-Speech Translation
Speech-to-speech translation combines machine translation with speech synthesis, introducing evaluation challenges not present in either task alone. How to automatically evaluate speech-to-speech translation is an open q…
Machine TranslationOpen-Ended Question AnsweringSpeech SynthesisSpeech-to-Speech Translation+1Findings of the IWSLT 2022 Evaluation Campaign
The evaluation campaign of the 19th International Conference on Spoken Language Translation featured eight shared tasks: (i) Simultaneous speech translation, (ii) Offline speech translation, (iii) Speech to speech transl…
Speech-to-Speech TranslationTranslationFINDINGS OF THE IWSLT 2020 EVALUATION CAMPAIGN
The evaluation campaign of the International Conference on Spoken Language Translation (IWSLT 2020) featured this year six challenge tracks: (i) Simultaneous speech translation, (ii) Video speech translation, (iii) Offli…
TranslationFINDINGS OF THE IWSLT 2021 EVALUATION CAMPAIGN
The evaluation campaign of the International Conference on Spoken Language Translation (IWSLT 2021) featured this year four shared tasks: (i) Simultaneous speech translation, (ii) Offline speech translation, (iii) Multil…
TranslationBLASER: A Text-Free Speech-to-Speech Translation Evaluation Metric
End-to-End speech-to-speech translation (S2ST) is generally evaluated with text-based metrics. This means that generated speech has to be automatically transcribed, making the evaluation dependent on the availability and…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)SentenceSpeech Recognition+2