Dscorer: A Fast Evaluation Metric for Discourse Representation Structure Parsing
Discourse representation structures (DRSs) are scoped semantic representations for texts of arbitrary length. Evaluating the accuracy of predicted DRSs plays a key role in developing semantic parsers and improving their performance. DRSs are typically visualized as boxes which are not straightforward to process automatically. Counter transforms DRSs to clauses and measures clause overlap by searching for variable mappings between two DRSs. However, this metric is computationally costly (with respect to memory and CPU time) and does not scale with longer texts. We introduce Dscorer, an efficient new metric which converts box-style DRSs to graphs and then measures the overlap of n-grams. Experiments show that Dscorer computes accuracy scores that are correlated with Counter at a fraction of the time.
Code (0)
등록된 구현이 없습니다.
Tasks
CPUSimilar Papers 제목 키워드 기반
Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning
This work presents a novel objective function for the unsupervised training of neural network sentence encoders. It exploits signals from paragraph-level discourse coherence to train these models to understand text. Our …
Representation LearningSentenceDiscoTK: Using Discourse Structure for Machine Translation Evaluation
We present novel automatic metrics for machine translation evaluation that use discourse structure and convolution kernels to compare the discourse tree of an automatic translation with that of the human reference. We ex…
Machine TranslationTranslationHow much progress have we made on RST discourse parsing? A replication study of recent results on the RST-DT
This article evaluates purported progress over the past years in RST discourse parsing. Several studies report a relative error reduction of 24 to 51{\%} on all metrics that authors attribute to the introduction of distr…
AttributeDiscourse ParsingDEPTH: Discourse Education through Pre-Training Hierarchically
Language Models (LMs) struggle with linguistic understanding at the discourse level, even though discourse patterns such as coherence, cohesion, and narrative flow are prevalent in their pre-training data. To improve the…
DecoderNatural Language UnderstandingSentenceSegmentation evaluation metrics, a comparison grounded on prosodic and discourse units
Knowledge on evaluation metrics and best practices of using them have improved fast in the recent years Fort et al. (2012). However, the advances concern mostly evaluation of classification related tasks. Segmentation ta…
Discourse SegmentationInformation RetrievalSegmentation