SUM-QE: a BERT-based Summary Quality Estimation Model

2019-11-01 · IJCNLP 2019 11 · Stratos Xenouleas, Prodromos Malakasiotis, Marianna Apidianaki, Ion Androutsopoulos

We propose SUM-QE, a novel Quality Estimation model for summarization based on BERT. The model addresses linguistic quality aspects that are only indirectly captured by content-based approaches to summary evaluation, without involving comparison with human references. SUM-QE achieves very high correlations with human ratings, outperforming simpler models addressing these linguistic aspects. Predictions of the SUM-QE model can be used for system development, and to inform users of the quality of automatically produced summaries and other types of generated text.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Weight Decay 설명 없음

Residual Connection 설명 없음

Adam 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

SumQE: a BERT-based Summary Quality Estimation Model

2019-09-02 · Stratos Xenouleas, Prodromos Malakasiotis, Marianna Apidianaki, Ion Androutsopoulos

We propose SumQE, a novel Quality Estimation model for summarization based on BERT. The model addresses linguistic quality aspects that are only indirectly captured by content-based approaches to summary evaluation, with…

model

Fill in the BLANC: Human-free quality estimation of document summaries

2020-02-23 · EMNLP (Eval4NLP) 2020 11 · Oleg Vasilyev, Vedant Dharnidharka, John Bohannon

We present BLANC, a new approach to the automatic estimation of document summary quality. Our goal is to measure the functional performance of a summary with an objective, reproducible, and fully automated method. Our ap…

Language Modeling Language Modelling

Understanding the Extent to which Content Quality Metrics Measure the Information Quality of Summaries

2021-11-01 · CoNLL (EMNLP) 2021 11 · Daniel Deutsch, Dan Roth

Reference-based metrics such as ROUGE or BERTScore evaluate the content quality of a summary by comparing the summary to a reference. Ideally, this comparison should measure the summary’s information quality by calculati…

Question Answering

DocAsRef: An Empirical Study on Repurposing Reference-Based Summary Quality Metrics Reference-Freely

2022-12-20 · Forrest Sheng Bao, Ruixuan Tu, Ge Luo, Yinfei Yang 외

Automated summary quality assessment falls into two categories: reference-based and reference-free. Reference-based metrics, historically deemed more accurate due to the additional information provided by human-written r…

Understanding the Extent to which Summarization Evaluation Metrics Measure the Information Quality of Summaries

2020-10-23 · Daniel Deutsch, Dan Roth