Designing and Evaluating a Reliable Corpus of Web Genres via Crowd-Sourcing
Research in Natural Language Processing often relies on a large collection of manually annotated documents. However, currently there is no reliable genre-annotated corpus of web pages to be employed in Automatic Genre Identification (AGI). In AGI, documents are classified based on their genres rather than their topics or subjects. The major shortcoming of available web genre collections is their relatively low inter-coder agreement. Reliability of annotated data is an essential factor for reliability of the research result. In this paper, we present the first web genre corpus which is reliably annotated. We developed precise and consistent annotation guidelines which consist of well-defined and well-recognized categories. For annotating the corpus, we used crowd-sourcing which is a novel approach in genre annotation. We computed the overall as well as the individual categories{'} chance-corrected inter-annotator agreement. The results show that the corpus has been annotated reliably.
Code (0)
등록된 구현이 없습니다.
Tasks
Information RetrievalMachine TranslationSimilar Papers 제목 키워드 기반
The NewSoMe Corpus: A Unifying Opinion Annotation Framework across Genres and in Multiple Languages
We present the NewSoMe (News and Social Media) Corpus, a set of subcorpora with annotations on opinion expressions across genres (news reports, blogs, product reviews and tweets) and covering multiple languages (English,…
Information RetrievalOpinion MiningCorpus Annotation through Crowdsourcing: Towards Best Practice Guidelines
Crowdsourcing is an emerging collaborative approach that can be used for the acquisition of annotated corpora and a wide range of other linguistic resources. Although the use of this approach is intensifying in all its k…
Domain AdaptationNatural Language InferenceOpinion MiningQuestion Answering+1DiscoGeM: A Crowdsourced Corpus of Genre-Mixed Implicit Discourse Relations
We present DiscoGeM, a crowdsourced corpus of 6,505 implicit discourse relations from three genres: political speech, literature, and encyclopedic texts. Each instance was annotated by 10 crowd workers. Various label agg…
RelationRelation ClassificationThe Search for Agreement on Logical Fallacy Annotation of an Infodemic
We evaluate an annotation schema for labeling logical fallacy types, originally developed for a crowd-sourcing annotation paradigm, now using an annotation paradigm of two trained linguist annotators. We apply the schema…
Logical FallaciesA good space: Lexical predictors in word space evaluation
Vector space models benefit from using an outside corpus to train the model. It is, however, unclear what constitutes a good training corpus. We have investigated the effect on summary quality when using various language…
Text CategorizationWord Sense Disambiguation