How Lexical Gold Standards Have Effects On The Usefulness Of Text Analysis Tools For Digital Scholarship
This paper describes how the current lexical similarity and analogy gold standards are built to conform to certain ideas about what the models they are designed to evaluate are used for. Topical relevance has always been the most important target notion for information access tools and related language technology technologies, and while this has proven a useful starting point for much of what information technology is used for, it does not always align well with other uses to which technologies are being put, most notably use cases from digital scholarship in the humanities or social sciences. This paper argues for more systematic formulation of requirements from the digital humanities and social sciences and more explicit description of the assumptions underlying model design.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
A Framework for Evaluation of Machine Reading Comprehension Gold Standards
Machine Reading Comprehension (MRC) is the task of answering a question over a paragraph of text. While neural MRC systems gain popularity and achieve noticeable performance, issues are being raised with the methodology …
Machine Reading ComprehensionReading ComprehensionSimulating Lexical Semantic Change from Sense-Annotated Data
We present a novel procedure to simulate lexical semantic change from synchronic sense-annotated data, and demonstrate its usefulness for assessing lexical semantic change detection models. The induced dataset represents…
Change DetectionSemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection
Lexical Semantic Change detection, i.e., the task of identifying words that change meaning over time, is a very active research area, with applications in NLP, lexicography, and linguistics. Evaluation is currently the m…
Change DetectionAccessing and standardizing Wiktionary lexical entries for the translation of labels in Cultural Heritage taxonomies
We describe the usefulness of Wiktionary, the freely available web-based lexical resource, in providing multilingual extensions to catalogues that serve content-based indexing of folktales and related narratives. We deve…
TranslationOdi et Amo. Creating, Evaluating and Extending Sentiment Lexicons for Latin.
Sentiment lexicons are essential for developing automatic sentiment analysis systems, but the resources currently available mostly cover modern languages. Lexicons for ancient languages are few and not evaluated with hig…
Cultural Vocal Bursts Intensity PredictionSentiment Analysis