MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
A robust evaluation metric has a profound impact on the development of text generation systems. A desirable metric compares system output against references based on their semantics rather than surface forms. In this paper we investigate strategies to encode system and reference texts to devise a metric that shows a high correlation with human judgment of text quality. We validate our new metric, namely MoverScore, on a number of text generation tasks including summarization, machine translation, image captioning, and data-to-text generation, where the outputs are produced by a variety of neural and non-neural systems. Our findings suggest that metrics combining contextualized representations with a distance measure perform the best. Such metrics also demonstrate strong generalization capability across tasks. For ease-of-use we make our metrics available as web service.
Code (4)
Tasks
Data-to-Text GenerationImage CaptioningMachine TranslationText GenerationTranslationSimilar Papers 제목 키워드 기반
Automatic Text Evaluation through the Lens of Wasserstein Barycenters
A new metric \texttt{BaryScore} to evaluate text generation based on deep contextualized embeddings e.g., BERT, Roberta, ELMo) is introduced. This metric is motivated by a new framework relying on optimal transport tools…
Image CaptioningMachine TranslationText GenerationText Summarization+1Sentence Pair Embeddings Based Evaluation Metric for Abstractive and Extractive Summarization
The development of an automatic evaluation metric remains an open problem in text generation. Widely used evaluation metrics, like ROUGE and BLEU, are based on exact word matching and fail to capture semantic similarity.…
Extractive SummarizationNatural Language InferenceSemantic SimilaritySemantic Textual Similarity+5Evaluating the Underlying Gender Bias in Contextualized Word Embeddings
Gender bias is highly impacting natural language processing applications. Word embeddings have clearly been proven both to keep and amplify gender biases that are present in current data sources. Recently, contextualized…
SentenceWord EmbeddingsEvaluating Contextualized Embeddings on 54 Languages in POS Tagging, Lemmatization and Dependency Parsing
We present an extensive evaluation of three recently proposed methods for contextualized embeddings on 89 corpora in 54 languages of the Universal Dependencies 2.3 in three tasks: POS tagging, lemmatization, and dependen…
Dependency ParsingLemmatizationPOSPOS Tagging+1UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation
Despite the success of existing referenced metrics (e.g., BLEU and MoverScore), they correlate poorly with human judgments for open-ended text generation including story or dialog generation because of the notorious one-…
Story GenerationText Generation