Exploring Prediction Uncertainty in Machine Translation Quality Estimation
Machine Translation Quality Estimation is a notoriously difficult task, which lessens its usefulness in real-world translation environments. Such scenarios can be improved if quality predictions are accompanied by a measure of uncertainty. However, models in this task are traditionally evaluated only in terms of point estimate metrics, which do not take prediction uncertainty into account. We investigate probabilistic methods for Quality Estimation that can provide well-calibrated uncertainty estimates and evaluate them in terms of their full posterior predictive distributions. We also show how this posterior information can be useful in an asymmetric risk scenario, which aims to capture typical situations in translation workflows.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationPredictionTranslationSimilar Papers 제목 키워드 기반
Uncertainty-Aware Machine Translation Evaluation
Several neural-based metrics have been recently proposed to evaluate machine translation quality. However, all of them resort to point estimates, which provide limited information at segment level. This is made worse as …
Machine TranslationTranslationConformalizing Machine Translation Evaluation
Several uncertainty estimation methods have been recently proposed for machine translation evaluation. While these methods can provide a useful indication of when not to trust model predictions, we show in this paper tha…
Conformal PredictionMachine TranslationPredictionTranslationEvaluating Machine Translation Quality with Conformal Predictive Distributions
This paper presents a new approach for assessing uncertainty in machine translation by simultaneously evaluating translation quality and providing a reliable confidence score. Our approach utilizes conformal predictive d…
Machine TranslationPrediction IntervalsTranslationExploring Consensus in Machine Translation for Quality Estimation
Disentangling Uncertainty in Machine Translation Evaluation
Trainable evaluation metrics for machine translation (MT) exhibit strong correlation with human judgements, but they are often hard to interpret and might produce unreliable scores under noisy or out-of-domain data. Rece…
Machine TranslationPredictionTranslationUncertainty Quantification