Fine-grained evaluation of German-English Machine Translation based on a Test Suite
We present an analysis of 16 state-of-the-art MT systems on German-English based on a linguistically-motivated test suite. The test suite has been devised manually by a team of language professionals in order to cover a broad variety of linguistic phenomena that MT often fails to translate properly. It contains 5,000 test sentences covering 106 linguistic phenomena in 14 categories, with an increased focus on verb tenses, aspects and moods. The MT outputs are evaluated in a semi-automatic way through regular expressions that focus only on the part of the sentence that is relevant to each phenomenon. Through our analysis, we are able to compare systems based on their performance on these categories. Additionally, we reveal strengths and weaknesses of particular systems and we identify grammatical phenomena where the overall performance of MT is relatively low.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationSentenceTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Linguistic Evaluation for the 2021 State-of-the-art Machine Translation Systems for German to English and English to German
We are using a semi-automated test suite in order to provide a fine-grained linguistic evaluation for state-of-the-art machine translation systems. The evaluation includes 18 German to English and 18 English to German sy…
Machine TranslationTranslationGuiding Large Language Models to Post-Edit Machine Translation with Error Annotations
Machine Translation (MT) remains one of the last NLP tasks where large language models (LLMs) have not yet replaced dedicated supervised systems. This work exploits the complementary strengths of LLMs and supervised MT b…
Machine TranslationTranslationA Linguistically Motivated Test Suite to Semi-Automatically Evaluate German–English Machine Translation Output
This paper presents a fine-grained test suite for the language pair German–English. The test suite is based on a number of linguistically motivated categories and phenomena and the semi-automatic evaluation is carried ou…
Machine TranslationZero-Shot to Full-Resource: Cross-lingual Transfer Strategies for Aspect-Based Sentiment Analysis
Aspect-based Sentiment Analysis (ABSA) extracts fine-grained opinions toward specific aspects within text but remains largely English-focused despite major advances in transformer-based and instruction-tuned models. This…
Cross-Lingual TransferMachine TranslationSentiment AnalysisA Multilingual Evaluation of NER Robustness to Adversarial Inputs
Adversarial evaluations of language models typically focus on English alone. In this paper, we performed a multilingual evaluation of Named Entity Recognition (NER) in terms of its robustness to small perturbations in th…
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)NER