A Linguistically Motivated Test Suite to Semi-Automatically Evaluate German–English Machine Translation Output
This paper presents a fine-grained test suite for the language pair German–English. The test suite is based on a number of linguistically motivated categories and phenomena and the semi-automatic evaluation is carried out with regular expressions. We describe the creation and implementation of the test suite in detail, providing a full list of all categories and phenomena. Furthermore, we present various exemplary applications of our test suite that have been implemented in the past years, like contributions to the Conference of Machine Translation, the usage of the test suite and MT outputs for quality estimation, and the expansion of the test suite to the language pair Portuguese–English. We describe how we tracked the development of the performance of various systems MT systems over the years with the help of the test suite and which categories and phenomena are prone to resulting in MT errors. For the first time, we also make a large part of our test suite publicly available to the research community.
Code (1)
Tasks
Machine TranslationSimilar Papers 제목 키워드 기반
Fine-grained evaluation of Quality Estimation for Machine translation based on a linguistically-motivated Test Suite
We present an alternative method of evaluating Quality Estimation systems, which is based on a linguistically-motivated Test Suite. We create a test-set consisting of 14 linguistic error categories and we gather for each…
Machine TranslationTranslationFine-grained evaluation of Quality Estimation for Machine translation based on a linguistically motivated Test Suite
Fine-grained evaluation of German-English Machine Translation based on a Test Suite
We present an analysis of 16 state-of-the-art MT systems on German-English based on a linguistically-motivated test suite. The test suite has been devised manually by a team of language professionals in order to cover a …
Machine TranslationSentenceTranslationDeep Semi-Supervised Learning with Linguistically Motivated Sequence Labeling Task Hierarchies
In this paper we present a novel Neural Network algorithm for conducting semi-supervised learning for sequence labeling tasks arranged in a linguistically motivated hierarchy. This relationship is exploited to regularise…
ChunkingLinguistic Features for Readability Assessment
Readability assessment aims to automatically classify text by the level appropriate for learning readers. Traditional approaches to this task utilize a variety of linguistically motivated features paired with simple mach…
Deep LearningText Classification