Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference
A major challenge in evaluating data-to-text (D2T) generation is measuring the semantic accuracy of the generated text, i.e. checking if the output text contains all and only facts supported by the input data. We propose a new metric for evaluating the semantic accuracy of D2T generation based on a neural model pretrained for natural language inference (NLI). We use the NLI model to check textual entailment between the input data and the output text in both directions, allowing us to reveal omissions or hallucinations. Input data are converted to text for NLI using trivial templates. Our experiments on two recent D2T datasets show that our metric can achieve high accuracy in identifying erroneous system outputs.
Code (1)
Tasks
Data-to-Text GenerationNatural Language InferenceText GenerationSimilar Papers 제목 키워드 기반
Generation Challenges: Results of the Accuracy Evaluation Shared Task
The Shared Task on Evaluating Accuracy focused on techniques (both manual and automatic) for evaluating the factual accuracy of texts produced by neural NLG systems, in a sports-reporting domain. Four teams submitted eva…
CSEval: A Framework for Evaluating Clinical Semantics in Text-to-Image Generation
Text-to-image generation has been increasingly applied in medical domains for various purposes such as data augmentation and education. Evaluating the quality and clinical reliability of these generated images is essenti…
Text-to-Image GenerationData AugmentationShared Task on Evaluating Accuracy in Natural Language Generation
We propose a shared task on methodologies and algorithms for evaluating the accuracy of generated texts. Participants will measure the accuracy of basketball game summaries produced by NLG systems from basketball box sco…
Text GenerationT2VWorldBench: A Benchmark for Evaluating World Knowledge in Text-to-Video Generation
Text-to-video (T2V) models have shown remarkable performance in generating visually reasonable scenes, while their capability to leverage world knowledge for ensuring semantic consistency and factual accuracy remains lar…
Text-to-Video GenerationText-in-Context: Token-Level Error Detection for Table-to-Text Generation
We present our Charles-UPF submission for the Shared Task on Evaluating Accuracy in Generated Texts at INLG 2021. Our system can detect the errors automatically using a combination of a rule-based natural language genera…
Language ModelingLanguage ModellingSemantic SimilaritySemantic Textual Similarity+3