SentSim: Crosslingual Semantic Evaluation of Machine Translation
Machine translation (MT) is currently evaluated in one of two ways: in a monolingual fashion, by comparison with the system output to one or more human reference translations, or in a trained crosslingual fashion, by building a supervised model to predict quality scores from human-labeled data. In this paper, we propose a more cost-effective, yet well performing unsupervised alternative SentSim: relying on strong pretrained multilingual word and sentence representations, we directly compare the source with the machine translated sentence, thus avoiding the need for both reference translations and labelled training data. The metric builds on state-of-the-art embedding-based approaches {--} namely BERTScore and Word Mover{'}s Distance {--} by incorporating a notion of sentence semantic similarity. By doing so, it achieves better correlation with human scores on different datasets. We show that it outperforms these and other metrics in the standard monolingual setting (MT-reference translation), a well as in the source-MT bilingual setting, where it performs on par with glass-box approaches to quality estimation that rely on MT model information.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationSemantic SimilaritySemantic Textual SimilaritySentenceTranslationSimilar Papers 제목 키워드 기반
Fully Unsupervised Crosslingual Semantic Textual Similarity Metric Based on BERT for Identifying Parallel Data
We present a fully unsupervised crosslingual semantic textual similarity (STS) metric, based on contextual embeddings extracted from BERT {--} Bidirectional Encoder Representations from Transformers (Devlin et al., 2019)…
Machine TranslationNatural Language UnderstandingSemantic Textual SimilaritySTS+1Assessing Crosslingual Discourse Relations in Machine Translation
In an attempt to improve overall translation quality, there has been an increasing focus on integrating more linguistic elements into Machine Translation (MT). While significant progress has been achieved, especially rec…
Machine TranslationTranslationBootstrapping a Crosslingual Semantic Parser
Recent progress in semantic parsing scarcely considers languages other than English but professional translation can be prohibitively expensive. We adapt a semantic parser trained on a single language, such as English, t…
Machine TranslationSemantic ParsingTranslation