As Easy as 1, 2, 3: Behavioural Testing of NMT Systems for Numerical Translation
Mistranslated numbers have the potential to cause serious effects, such as financial loss or medical misinformation. In this work we develop comprehensive assessments of the robustness of neural machine translation systems to numerical text via behavioural testing. We explore a variety of numerical translation capabilities a system is expected to exhibit and design effective test examples to expose system underperformance. We find that numerical mistranslation is a general issue: major commercial systems and state-of-the-art research models fail on many of our test examples, for high- and low-resource languages. Our tests reveal novel errors that have not previously been reported in NMT systems, to the best of our knowledge. Lastly, we discuss strategies to mitigate numerical mistranslation.
Code (1)
Tasks
Machine TranslationMisinformationNMTTranslationSimilar Papers 제목 키워드 기반
The Behavioural Translation Style Space: Towards simulating the temporal dynamics of affect, behaviour, and cognition in human translation production
The paper introduces a novel behavioural translation style space (BTSS) that describes possible behavioural translation patterns. The suggested BTSS is organized as a hierarchical structure that entails various embedded …
A Behaviour and Disease Model of Testing and Isolation
There has been interest in the interactions between infectious disease dynamics and behaviour for most of the history of mathematical epidemiology. This has included consideration of which mathematical models best captur…
EpidemiologyMTUOC: easy and free integration of NMT systems in professional translation environments
In this paper the MTUOC project, aiming to provide an easy integration of neural and statistical machine translation systems, is presented. Almost all the required software to train and use neural and statistical MT syst…
Machine TranslationNMTTranslationTowards General Error Diagnosis via Behavioral Testing in Machine Translation
Behavioral testing offers a crucial means of diagnosing linguistic errors and assessing capabilities of NLP models. However, applying behavioral testing to machine translation (MT) systems is challenging as it generally …
Machine TranslationTranslationSWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?
Modern software systems accumulate technical debt over decades of development, which makes migration expensive and largely manual. As coding agents become increasingly capable at bug fixing, can they autonomously perform…