NMTScore: A Multilingual Analysis of Translation-based Text Similarity Measures
Being able to rank the similarity of short text segments is an interesting bonus feature of neural machine translation. Translation-based similarity measures include direct and pivot translation probability, as well as translation cross-likelihood, which has not been studied so far. We analyze these measures in the common framework of multilingual NMT, releasing the NMTScore library (available at https://github.com/ZurichNLP/nmtscore). Compared to baselines such as sentence embeddings, translation-based measures prove competitive in paraphrase identification and are more robust against adversarial or multilingual input, especially if proper normalization is applied. When used for reference-based evaluation of data-to-text generation in 2 tasks and 17 languages, translation-based measures show a relatively high correlation to human judgments.
Code (1)
Tasks
Data-to-Text GenerationMachine TranslationNMTParaphrase IdentificationSentenceSentence EmbeddingsText Generationtext similarityTranslationSimilar Papers 제목 키워드 기반
Towards a Deep Understanding of Multilingual End-to-End Speech Translation
In this paper, we employ Singular Value Canonical Correlation Analysis (SVCCA) to analyze representations learnt in a multilingual end-to-end speech translation model trained over 22 languages. SVCCA enables us to estima…
Machine TranslationTranslationEliciting the Translation Ability of Large Language Models via Multilingual Finetuning with Translation Instructions
Large-scale Pretrained Language Models (LLMs), such as ChatGPT and GPT4, have shown strong abilities in multilingual translations, without being explicitly trained on parallel corpora. It is interesting how the LLMs obta…
Language ModelingLanguage ModellingTranslationSemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation
Semantic Textual Similarity (STS) measures the meaning similarity of sentences. Applications include machine translation (MT), summarization, generation, question answering (QA), short answer grading, semantic search, di…
Machine TranslationNatural Language InferenceQuestion AnsweringSemantic Textual Similarity+3Improving Multilingual Semantic Textual Similarity with Shared Sentence Encoder for Low-resource Languages
Measuring the semantic similarity between two sentences (or Semantic Textual Similarity - STS) is fundamental in many NLP applications. Despite the remarkable results in supervised settings with adequate labeling, little…
Machine TranslationSemantic SimilaritySemantic Textual SimilaritySentence+2Investigating Multilingual NMT Representations at Scale
Multilingual Neural Machine Translation (NMT) models have yielded large empirical success in transfer learning settings. However, these black-box representations are poorly understood, and their mode of transfer remains …
Cross-Lingual TransferMachine TranslationNMTTransfer Learning+1