SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Window
Reference-based metrics that operate at the sentence-level typically outperform quality estimation metrics, which have access only to the source and system output. This is unsurprising, since references resolve ambiguities that may be present in the source. In this paper, we investigate whether additional source context can effectively substitute for a reference. We present a metric named SLIDE (SLIding Document Evaluator), which operates on blocks of sentences. SLIDE leverages a moving window that slides over each document in the test set, feeding each chunk of sentences into an unmodified, off-the-shelf quality estimation model. We find that SLIDE obtains significantly higher pairwise system accuracy than its sentence-level baseline, in some cases even eliminating the gap with reference-base metrics. This suggests that source context may provide the same information as a human reference in disambiguating source ambiguities. This finding is especially pertinent for reference-free document-level evaluation, wherein SLIDE could provide higher-quality pairwise system assessments while only requiring document boundary annotations.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationSentenceSimilar Papers 제목 키워드 기반
KG-BERTScore: Incorporating Knowledge Graph into BERTScore for Reference-Free Machine Translation Evaluation
BERTScore is an effective and robust automatic metric for referencebased machine translation evaluation. In this paper, we incorporate multilingual knowledge graph into BERTScore and propose a metric named KG-BERTScore, …
Machine TranslationTranslationAlign-then-Slide: A complete evaluation framework for Ultra-Long Document-Level Machine Translation
Large language models (LLMs) have ushered in a new era for document-level machine translation (\textit{doc}-mt), yet their whole-document outputs challenge existing evaluation methods that assume sentence-by-sentence ali…
Machine TranslationMT-Ranker: Reference-free machine translation evaluation by inter-system ranking
Traditionally, Machine Translation (MT) Evaluation has been treated as a regression problem -- producing an absolute translation-quality score. This approach has two limitations: i) the scores lack interpretability, and …
Machine TranslationNatural Language InferenceSentenceTranslationOn the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation
Evaluation of cross-lingual encoders is usually performed either via zero-shot cross-lingual transfer in supervised downstream tasks or via unsupervised cross-lingual textual similarity. In this paper, we concern ourselv…
Cross-Lingual TransferLanguage ModelingLanguage ModellingMachine Translation+4Assessing Reference-Free Peer Evaluation for Machine Translation
Reference-free evaluation has the potential to make machine translation evaluation substantially more scalable, allowing us to pivot easily to new languages or domains. It has been recently shown that the probabilities g…
Machine TranslationTranslation