Towards General Error Diagnosis via Behavioral Testing in Machine Translation
Behavioral testing offers a crucial means of diagnosing linguistic errors and assessing capabilities of NLP models. However, applying behavioral testing to machine translation (MT) systems is challenging as it generally requires human efforts to craft references for evaluating the translation quality of such systems on newly generated test cases. Existing works in behavioral testing of MT systems circumvent this by evaluating translation quality without references, but this restricts diagnosis to specific types of errors, such as incorrect translation of single numeric or currency words. In order to diagnose general errors, this paper proposes a new Bilingual Translation Pair Generation based Behavior Testing (BTPGBT) framework for conducting behavioral testing of MT systems. The core idea of BTPGBT is to employ a novel bilingual translation pair generation (BTPG) approach that automates the construction of high-quality test cases and their pseudoreferences. Experimental results on various MT systems demonstrate that BTPGBT could provide comprehensive and accurate behavioral testing results for general error diagnosis, which further leads to several insightful findings. Our code and data are available at https: //github.com/wujunjie1998/BTPGBT.
Code (1)
Tasks
Machine TranslationTranslationSimilar Papers 제목 키워드 기반
Beyond NDCG: behavioral testing of recommender systems with RecList
As with most Machine Learning systems, recommender systems are typically evaluated through performance metrics computed over held-out data points. However, real-world behavior is undoubtedly nuanced: ad hoc error analysi…
Recommendation SystemsAutomating Behavioral Testing in Machine Translation
Behavioral testing in NLP allows fine-grained evaluation of systems by examining their linguistic capabilities through the analysis of input-output behavior. Unfortunately, existing work on behavioral testing in Machine …
Machine TranslationTranslationSALTED: A Framework for SAlient Long-Tail Translation Error Detection
Traditional machine translation (MT) metrics provide an average measure of translation quality that is insensitive to the long tail of behavioral problems in MT. Examples include translation of numbers, physical units, d…
Machine TranslationNMTSentenceTranslationDigitally-Enhanced Dog Behavioral Testing: Getting Help from the Machine
The assessment of behavioral traits in dogs is a well-studied challenge due to its many practical applications such as selection for breeding, prediction of working aptitude, chances of being adopted, etc. Most methods f…
Clustering