Does Summary Evaluation Survive Translation to Other Languages?
The creation of a quality summarization dataset is an expensive, time-consuming effort, requiring the production and evaluation of summaries by both trained humans and machines. If such effort is made in one language, it would be beneficial to be able to use it in other languages without repeating human annotations. To investigate how much we can trust machine translation of such a dataset, we translate the English dataset SummEval to seven languages and compare performance across automatic evaluation measures. We explore equivalence testing as the appropriate statistical paradigm for evaluating correlations between human and automated scoring of summaries. While we find some potential for dataset reuse in languages similar to the source, most summary evaluation methods are not found to be statistically equivalent across translations.
Code (1)
Tasks
Machine TranslationTranslationSimilar Papers 제목 키워드 기반
Does Summary Evaluation Survive Translation to Other Languages?
The creation of a quality summarization dataset is an expensive, time-consuming effort, requiring the production and evaluation of summaries by both trained humans and machines. The returns to such an effort would increa…
Machine TranslationTranslationMachine Translation for Multilingual Summary Content Evaluation
Breeding Machine Translations: Evolutionary approach to survive and thrive in the world of automated evaluation
We propose a genetic algorithm (GA) based method for modifying n-best lists produced by a machine translation (MT) system. Our method offers an innovative approach to improving MT quality and identifying weaknesses in ev…
Machine TranslationTranslationPyramid-based Summary Evaluation Using Abstract Meaning Representation
We propose a novel metric for evaluating summary content coverage. The evaluation framework follows the Pyramid approach to measure how many summarization content units, considered important by human annotators, are cont…
Abstract Meaning RepresentationGraph SimilaritySentenceHEVAL: Yet Another Human Evaluation Metric
Machine translation evaluation is a very important activity in machine translation development. Automatic evaluation metrics proposed in literature are inadequate as they require one or more human reference translations …
Machine TranslationTranslation