Just Ask! Evaluating Machine Translation by Asking and Answering Questions
In this paper, we show that automatically-generated questions and answers can be used to evaluate the quality of Machine Translation (MT) systems. Building on recent work on the evaluation of abstractive text summarization, we propose a new metric for system-level MT evaluation, compare it with other state-of-the-art solutions, and show its robustness by conducting experiments for various MT directions.
Code (1)
Tasks
Abstractive Text SummarizationMachine TranslationText SummarizationTranslationSimilar Papers 제목 키워드 기반
What's Different between Visual Question Answering for Machine "Understanding" Versus for Accessibility?
In visual question answering (VQA), a machine must answer a question given an associated image. Recently, accessibility researchers have explored whether VQA can be deployed in a real-world setting where users with visua…
BenchmarkingQuestion AnsweringVisual Question AnsweringVisual Question Answering (VQA)On the Evaluation of Machine Translation n-best Lists
The standard machine translation evaluation framework measures the single-best output of machine translation systems. There are, however, many situations where n-best lists are needed, yet there is no established way of …
Machine TranslationTranslationvalidDo LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answering
Despite the steady progress in machine translation evaluation, existing automatic metrics struggle to capture how well meaning is preserved beyond sentence boundaries. We posit that reliance on a single intrinsic quality…
Machine TranslationQuestion AnsweringReading ComprehensionSentence+1Efficient Object-Level Visual Context Modeling for Multimodal Machine Translation: Masking Irrelevant Objects Helps Grounding
Visual context provides grounding information for multimodal machine translation (MMT). However, previous MMT models and probing studies on visual features suggest that visual information is less explored in MMT as it is…
Machine TranslationMultimodal Machine TranslationObjectTranslationFindings of the WMT 2019 Shared Tasks on Quality Estimation
We report the results of the WMT19 shared task on Quality Estimation, i.e. the task of predicting the quality of the output of machine translation systems given just the source text and the hypothesis translations. The t…
Machine TranslationSentenceTranslation