Semantic Similarity Based Evaluation for Abstractive News Summarization
ROUGE is a widely used evaluation metric in text summarization. However, it is not suitable for the evaluation of abstractive summarization systems as it relies on lexical overlap between the gold standard and the generated summaries. This limitation becomes more apparent for agglutinative languages with very large vocabularies and high type/token ratios. In this paper, we present semantic similarity models for Turkish and apply them as evaluation metrics for an abstractive summarization task. To achieve this, we translated the English STSb dataset into Turkish and presented the first semantic textual similarity dataset for Turkish as well. We showed that our best similarity models have better alignment with average human judgments compared to ROUGE in both Pearson and Spearman correlations.
Code (1)
Tasks
Abstractive Text SummarizationNews SummarizationSemantic SimilaritySemantic Textual SimilarityText SummarizationSimilar Papers 제목 키워드 기반
A Two-Phase Approach for Abstractive Podcast Summarization
Podcast summarization is different from summarization of other data formats, such as news, patents, and scientific papers in that podcasts are often longer, conversational, colloquial, and full of sponsorship and adverti…
DecoderSentenceSentence SimilarityVocal Bursts Valence PredictionEvent-based evaluation of abstractive news summarization
An abstractive summary of a news article contains its most important information in a condensed version. The evaluation of automatically generated summaries by generative language models relies heavily on human-authored …
Abstractive News Summarization based on Event Semantic Link Network
This paper studies the abstractive multi-document summarization for event-oriented news texts through event information extraction and abstract representation. Fine-grained event mentions and semantic relations between t…
Abstractive Text SummarizationDocument SummarizationMulti-Document SummarizationNews SummarizationFrom News to Summaries: Building a Hungarian Corpus for Extractive and Abstractive Summarization
Training summarization models requires substantial amounts of training data. However for less resourceful languages like Hungarian, openly available models and datasets are notably scarce. To address this gap our paper i…
Abstractive Text SummarizationExtractive SummarizationSentenceSentence SimilarityOverview of the VLSP 2022 -- Abmusu Shared Task: A Data Challenge for Vietnamese Abstractive Multi-document Summarization
This paper reports the overview of the VLSP 2022 - Vietnamese abstractive multi-document summarization (Abmusu) shared task for Vietnamese News. This task is hosted at the 9$^{th}$ annual workshop on Vietnamese Language …
Document SummarizationMulti-Document SummarizationNews Summarization