MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers
As major progress is made in open-ended text generation, measuring how close machine-generated text is to human language remains a critical open problem. We introduce MAUVE, a comparison measure for open-ended text generation, which directly compares the learnt distribution from a text generation model to the distribution of human-written text using divergence frontiers. MAUVE scales up to modern text generation models by computing information divergences in a quantized embedding space. Through an extensive empirical study on three open-ended generation tasks, we find that MAUVE identifies known properties of generated text, scales naturally with model size, and correlates with human judgments, with fewer restrictions than existing distributional evaluation metrics.
Code (5)
Tasks
Text GenerationSimilar Papers 제목 키워드 기반
MAUVE Scores for Generative Models: Theory and Practice
Generative artificial intelligence has made significant strides, producing text indistinguishable from human prose and remarkably photorealistic images. Automatically measuring how close the generated data distribution i…
QuantizationOn the Usefulness of Embeddings, Clusters and Strings for Text Generator Evaluation
A good automatic evaluation metric for language generation ideally correlates highly with human judgements of text quality. Yet, there is a dearth of such metrics, which inhibits the rapid and efficient progress of langu…
Language ModellingText GenerationAn Empirical Study On Contrastive Search And Contrastive Decoding For Open-ended Text Generation
In the study, we empirically compare the two recently proposed decoding methods, i.e. Contrastive Search (CS) and Contrastive Decoding (CD), for open-ended text generation. The automatic evaluation results suggest that, …
DiversityText GenerationFrom Aesthetics to Human Preferences: Comparative Perspectives of Evaluating Text-to-Music Systems
Evaluating generative models remains a fundamental challenge, particularly when the goal is to reflect human preferences. In this paper, we use music generation as a case study to investigate the gap between automatic ev…
Music GenerationAdversarial Robustness of Neural-Statistical Features in Detection of Generative Transformers
The detection of computer-generated text is an area of rapidly increasing significance as nascent generative models allow for efficient creation of compelling human-like text, which may be abused for the purposes of spam…
Adversarial RobustnessAdversarial Text