Evaluation rules! On the use of grammars and rule-based systems for NLG evaluation
NLG researchers often use uncontrolled corpora to train and evaluate their systems, using textual similarity metrics, such as BLEU. This position paper argues in favour of two alternative evaluation strategies, using grammars or rule-based systems. These strategies are particularly useful to identify the strengths and weaknesses of different systems. We contrast our proposals with the (extended) WebNLG dataset, which is revealed to have a skewed distribution of predicates. We predict that this distribution affects the quality of the predictions for systems trained on this data. However, this hypothesis can only be thoroughly tested (without any confounds) once we are able to systematically manipulate the skewness of the data, using a rule-based approach.
Code (0)
등록된 구현이 없습니다.
Tasks
nlg evaluationPositionSimilar Papers 제목 키워드 기반
Toward Bayesian Synchronous Tree Substitution Grammars for Sentence Planning
Developing conventional natural language generation systems requires extensive attention from human experts in order to craft complex sets of sentence planning rules. We propose a Bayesian nonparametric approach to learn…
SentenceText GenerationAn Empirical Evaluation of Rule Extraction from Recurrent Neural Networks
Rule extraction from black-box models is critical in domains that require model validation before implementation, as can be the case in credit scoring and medical diagnosis. Though already a challenging problem in statis…
Medical DiagnosisTowards a classification of Lindenmayer systems
In this paper we will attempt to classify Lindenmayer systems based on properties of sets of rules and the kind of strings those rules generate. This classification will be referred to as a parametrization of the L-space…
ClassificationGeneral ClassificationReinforced Molecular Optimization with Neighborhood-Controlled Grammars
A major challenge in the pharmaceutical industry is to design novel molecules with specific desired properties, especially when the property evaluation is costly. Here, we propose MNCE-RL, a graph convolutional policy ne…
Graph GenerationMolecular Graph GenerationThe Generalized Smallest Grammar Problem
The Smallest Grammar Problem -- the problem of finding the smallest context-free grammar that generates exactly one given sequence -- has never been successfully applied to grammatical inference. We investigate the reaso…