High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics
In Neural Machine Translation, it is typically assumed that the sentence with the highest estimated probability should also be the translation with the highest quality as measured by humans. In this work, we question this assumption and show that model estimates and translation quality only vaguely correlate. We apply Minimum Bayes Risk (MBR) decoding on unbiased samples to optimize diverse automated metrics of translation quality as an alternative inference strategy to beam search. Instead of targeting the hypotheses with the highest model probability, MBR decoding extracts the hypotheses with the highest estimated quality. Our experiments show that the combination of a neural translation model with a neural reference-based metric, BLEURT, results in significant improvement in human evaluations. This improvement is obtained with translations different from classical beam-search output: these translations have much lower model likelihood and are less favored by surface metrics like BLEU.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationSentenceTranslationVocal Bursts Intensity PredictionSimilar Papers 제목 키워드 기반
High probability or low information? The probability–quality paradox in language generation
When generating natural language from neural probabilistic models, high probability does not always coincide with high quality. Rather, mode-seeking decoding methods can lead to incredibly unnatural language, while stoch…
Text GenerationCompressive neural representation of sparse, high-dimensional probabilities
This paper shows how sparse, high-dimensional probability distributions could be represented by neurons with exponential compression. The representation is a novel application of compressive sensing to sparse probability…
Compressive SensingVocal Bursts Intensity PredictionIn Their Own Words: Reasoning Traces Tailored for Small Models Make Them Better Reasoners
Transferring reasoning capabilities from larger language models to smaller ones through supervised fine-tuning often fails counterintuitively, with performance degrading despite access to high-quality teacher demonstrati…
On the Step Length Confounding in LLM Reasoning Data Selection
Large reasoning models have recently demonstrated strong performance on complex tasks that require long chain-of-thought reasoning, through supervised fine-tuning on large-scale and high-quality datasets. To construct su…
Geodesic Gramian Denoising Applied to the Images Contaminated With Noise Sampled From Diverse Probability Distributions
As quotidian use of sophisticated cameras surges, people in modern society are more interested in capturing fine-quality images. However, the quality of the images might be inferior to people's expectations due to the no…
Denoising