Perplexity from PLM Is Unreliable for Evaluating Text Quality
Recently, amounts of works utilize perplexity~(PPL) to evaluate the quality of the generated text. They suppose that if the value of PPL is smaller, the quality(i.e. fluency) of the text to be evaluated is better. However, we find that the PPL referee is unqualified and it cannot evaluate the generated text fairly for the following reasons: (i) The PPL of short text is larger than long text, which goes against common sense, (ii) The repeated text span could damage the performance of PPL, and (iii) The punctuation marks could affect the performance of PPL heavily. Experiments show that the PPL is unreliable for evaluating the quality of given text. Last, we discuss the key problems with evaluating text quality using language models.
Code (0)
등록된 구현이 없습니다.
Tasks
Common Sense ReasoningSimilar Papers 제목 키워드 기반
Mirostat: A Neural Text Decoding Algorithm that Directly Controls Perplexity
Neural text decoding is important for generating high-quality texts using language models. To generate high-quality text, popular decoding algorithms like top-k, top-p (nucleus), and temperature-based sampling truncate o…
Language ModelingLanguage ModellingRethinking Perplexity: Revealing the Impact of Input Length on Perplexity Evaluation in LLMs
Perplexity is a widely adopted metric for assessing the predictive quality of large language models (LLMs) and often serves as a reference metric for downstream evaluations. However, recent evidence shows that perplexity…
What is Wrong with Perplexity for Long-context Language Modeling?
Handling long-context inputs is crucial for large language models (LLMs) in tasks such as extended conversations, document summarization, and many-shot in-context learning. While recent approaches have extended the conte…
Document SummarizationIn-Context LearningLanguage ModelingLanguage Modelling+1DUEL: Exact Likelihood for Masked Diffusion via Deterministic Unmasking
Masked diffusion models (MDMs) generate text by iteratively selecting positions to unmask and then predicting tokens at those positions. Yet MDMs lack proper likelihood evaluation: the evidence lower bound (ELBO) is not …
OPERA: Aligning Open-Ended Reasoning via Objective Perplexity-based Reinforcement Learning
Reinforcement Learning (RL) has enabled LLMs to excel in objective reasoning tasks such as mathematics and code generation. However, applying RL to open-ended tasks, such as creative writing, remains challenging because …
Reinforcement LearningCode Generation