Do Text Simplification Systems Preserve Meaning? A Human Evaluation via Reading Comprehension

2023-12-15 · Sweta Agrawal, Marine Carpuat

Automatic text simplification (TS) aims to automate the process of rewriting text to make it easier for people to read. A pre-requisite for TS to be useful is that it should convey information that is consistent with the meaning of the original text. However, current TS evaluation protocols assess system outputs for simplicity and meaning preservation without regard for the document context in which output sentences occur and for how people understand them. In this work, we introduce a human evaluation framework to assess whether simplified texts preserve meaning using reading comprehension questions. With this framework, we conduct a thorough human evaluation of texts by humans and by nine automatic systems. Supervised systems that leverage pre-training knowledge achieve the highest scores on the reading comprehension (RC) tasks amongst the automatic controllable TS systems. However, even the best-performing supervised system struggles with at least 14% of the questions, marking them as "unanswerable'' based on simplified content. We further investigate how existing TS evaluation metrics and automatic question-answering systems approximate the human judgments we obtained.

📄 PDF Abstract BibTeX arXiv:2312.10126

Code (1)

sweta20/ats-eval 공식 구현

Tasks

Question Answering Reading Comprehension Text Simplification

Methods 이 논문이 사용한 방법론

TS Spatio-temporal features extraction that measure the stabilty. The proposed method is based on a compression algorithm named Run Length Encoding. The workflow of the method is…

Exploring Neural Text Simplification Models

2017-07-01 · ACL 2017 7 · Sergiu Nisioi, Sanja {\v{S}}tajner, Simone Paolo Ponzetto, Liviu P. Dinu

We present the first attempt at using sequence to sequence neural networks to model text simplification (TS). Unlike the previously proposed automated TS systems, our neural text simplification (NTS) systems are able to …

Lexical Simplification Machine Translation Text Simplification Word Embeddings

Text Simplification with Reinforcement Learning Using Supervised Rewards on Grammaticality, Meaning Preservation, and Simplicity

2020-12-01 · Asian Chapter of the Association for Computational Linguistics 2020 · Akifumi Nakamachi, Tomoyuki Kajiwara, Yuki Arase

We optimize rewards of reinforcement learning in text simplification using metrics that are highly correlated with human-perspectives. To address problems of exposure bias and loss-evaluation mismatch, text-to-text gener…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Text Generation+1

Reference-less Quality Estimation of Text Simplification Systems

2019-01-30 · WS 2018 11 · Louis Martin, Samuel Humeau, Pierre-Emmanuel Mazaré, Antoine Bordes 외

The evaluation of text simplification (TS) systems remains an open challenge. As the task has common points with machine translation (MT), TS is often evaluated using MT metrics such as BLEU. However, such metrics requir…

Machine Translation Sentence Text Simplification Translation

Sentence Simplification with Deep Reinforcement Learning

2017-03-31 · EMNLP 2017 9 · Xingxing Zhang, Mirella Lapata

Sentence simplification aims to make sentences easier to read and understand. Most recent approaches draw on insights from machine translation to learn simplification rewrites from monolingual corpora of complex and simp…

Decoder Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+5

Can Knowledge Graphs Simplify Text?

2023-08-14 · Anthony Colas, Haodi Ma, Xuanli He, Yang Bai 외

Knowledge Graph (KG)-to-Text Generation has seen recent improvements in generating fluent and informative sentences which describe a given KG. As KGs are widespread across multiple domains and contain important entity-re…

Descriptive KG-to-Text Generation Knowledge Graphs Text Generation+1