BERTScore: Evaluating Text Generation with BERT
We propose BERTScore, an automatic evaluation metric for text generation. Analogously to common metrics, BERTScore computes a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTScore correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task to show that BERTScore is more robust to challenging examples when compared to existing metrics.
Code (20)
Tasks
Image CaptioningMachine TranslationModel SelectionSentenceText GenerationTranslationSimilar Papers 제목 키워드 기반
CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code
Since the rise of neural natural-language-to-code models (NL->Code) that can generate long expressions and statements rather than a single next-token, one of the major problems has been reliably evaluating their generate…
Code GenerationFinNuE: Exposing the Risks of Using BERTScore for Numerical Semantic Evaluation in Finance
BERTScore has become a widely adopted metric for evaluating semantic similarity between natural language sentences. However, we identify a critical limitation: BERTScore exhibits low sensitivity to numerical variation, a…
Semantic SimilarityBERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generation
Automatic evaluation metrics are crucial to the development of generative systems. In recent years, pre-trained language model (PLM) based metrics, such as BERTScore, have been commonly adopted in various generation task…
FairnessLanguage ModelingLanguage ModellingText GenerationOARelatedWork: A Large-Scale Dataset of Related Work Sections with Full-texts from Open Access Sources
This paper introduces OARelatedWork, the first large-scale multi-document summarization dataset for related work generation containing whole related work sections and full-texts of cited papers. The dataset includes 94 4…
Document SummarizationExtractive SummarizationMulti-Document SummarizationEvaluating and Improving Factuality in Multimodal Abstractive Summarization
Current metrics for evaluating factuality for abstractive document summarization have achieved high correlations with human judgment, but they do not account for the vision modality and thus are not adequate for vision-a…
Abstractive Text SummarizationDocument Summarization