Crowdsourcing Discourse Relation Annotations by a Two-Step Connective Insertion Task
The perspective of being able to crowd-source coherence relations bears the promise of acquiring annotations for new texts quickly, which could then increase the size and variety of discourse-annotated corpora. It would also open the avenue to answering new research questions: Collecting annotations from a larger number of individuals per instance would allow to investigate the distribution of inferred relations, and to study individual differences in coherence relation interpretation. However, annotating coherence relations with untrained workers is not trivial. We here propose a novel two-step annotation procedure, which extends an earlier method by Scholman and Demberg (2017a). In our approach, coherence relation labels are inferred from connectives that workers insert into the text. We show that the proposed method leads to replicable coherence annotations, and analyse the agreement between the obtained relation labels and annotations from PDTB and RSTDT on the same texts.
Code (0)
등록된 구현이 없습니다.
Tasks
RelationVocal Bursts Valence PredictionSimilar Papers 제목 키워드 기반
Crowdsourcing discourse interpretations: On the influence of context and the reliability of a connective insertion task
Traditional discourse annotation tasks are considered costly and time-consuming, and the reliability and validity of these tasks is in question. In this paper, we investigate whether crowdsourcing can be used to obtain r…
RelationOn Crowdsourcing Task Design for Discourse Relation Annotation
Interpreting implicit discourse relations involves complex reasoning, requiring the integration of semantic cues with background knowledge, as overt connectives like because or then are absent. These relations often allo…
RelationDesign Choices in Crowdsourcing Discourse Relation Annotations: The Effect of Worker Selection and Training
Obtaining linguistic annotation from novice crowdworkers is far from trivial. A case in point is the annotation of discourse relations, which is a complicated task. Recent methods have obtained promising results by extra…
RelationTCL - a Lexicon of Turkish Discourse Connectives
It is known that discourse connectives are the most salient indicators of discourse relations. State-of-the-art parsers being developed to predict explicit discourse connectives exploit annotated discourse corpora but a …
Semi-automatic discourse annotation in a low-resource language: Developing a connective lexicon for Nigerian Pidgin
Cross-linguistic research on discourse structure and coherence marking requires discourse-annotated corpora and connective lexicons in a large number of languages. However, the availability of such resources is limited, …
Relation