Contrastive Conditioning for Assessing Disambiguation in MT: A Case Study of Distilled Bias
Lexical disambiguation is a major challenge for machine translation systems, especially if some senses of a word are trained less often than others. Identifying patterns of overgeneralization requires evaluation methods that are both reliable and scalable. We propose contrastive conditioning as a reference-free black-box method for detecting disambiguation errors. Specifically, we score the quality of a translation by conditioning on variants of the source that provide contrastive disambiguation cues. After validating our method, we apply it in a case study to perform a targeted evaluation of sequence-level knowledge distillation. By probing word sense disambiguation and translation of gendered occupation names, we show that distillation-trained models tend to overgeneralize more than other models with a comparable BLEU score. Contrastive conditioning thus highlights a side effect of distillation that is not fully captured by standard evaluation metrics. Code and data to reproduce our findings are publicly available.
Code (1)
Tasks
Knowledge DistillationMachine TranslationTranslationWord Sense DisambiguationSimilar Papers 제목 키워드 기반
Contrastive Entity Coreference and Disambiguation for Historical Texts
Massive-scale historical document collections are crucial for social science research. Despite increasing digitization, these documents typically lack unique cross-document identifiers for individuals mentioned within th…
Articlescoreference-resolutionCoreference ResolutionCross Document Coreference Resolution+1EMODIS: A Benchmark for Context-Dependent Emoji Disambiguation in Large Language Models
Large language models (LLMs) are increasingly deployed in real-world communication settings, yet their ability to resolve context-dependent ambiguity remains underexplored. In this work, we present EMODIS, a new benchmar…
Contrastive Self-Supervised Learning for Commonsense Reasoning
We propose a self-supervised method to solve Pronoun Disambiguation and Winograd Schema Challenge problems. Our approach exploits the characteristic structure of training corpora related to so-called "trigger" words, whi…
Self-Supervised LearningSimCLAD: A Simple Framework for Contrastive Learning of Acronym Disambiguation
Acronym disambiguation means finding the correct meaning of an ambiguous acronym from the dictionary in a given sentence, which is one of the key points for scientific document understanding (SDU@AAAI-22). Recently, many…
Contrastive Learningdocument understandingSentencePiCO+: Contrastive Label Disambiguation for Robust Partial Label Learning
Partial label learning (PLL) is an important problem that allows each training example to be labeled with a coarse candidate set, which well suits many real-world data annotation scenarios with label ambiguity. Despite t…
Contrastive LearningPartial Label LearningPICORepresentation Learning