Consensus Measures for Unstructured Biomedical Text Annotations
Biomedical literature is increasingly mined for knowledge beyond the questions it was written to answer. Because the target concepts are not known in advance, annotators prefer open-ended labels, whose agreement is hard to quantify. We study soft inter-rater reliability for annotators providing unstructured texts for biomedical annotation tasks. Synthetic experiments show that soft reliability can be quantified using a variety of semantic equivalence measures, and that the choice of measure affects failure modes of the estimation. Embeddings are scalable, but limited when differentiating similar but distinct concepts. Large language models are promising, but limited by scalability for estimating agreement by chance. Finally, we suggest measures based on natural language inference as a sensible compromise.
Code (0)
등록된 구현이 없습니다.
Tasks
Natural Language InferenceSimilar Papers 제목 키워드 기반
Aggregating and Predicting Sequence Labels from Crowd Annotations
Despite sequences being core to NLP, scant work has considered how to handle noisy sequence labels from multiple annotators for the same text. Given such annotations, we consider two complementary tasks: (1) aggregating …
Articlesnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)+1Gextext: Disease Network Extraction from Biomedical Literature
PURPOSE: We propose a fully unsupervised method to learn latent disease networks directly from unstructured biomedical text corpora. This method addresses current challenges in unsupervised knowledge extraction, such as …
Semantic SimilaritySemantic Textual SimilarityPlausibility-Driven Prioritization of Candidate Biomedical Annotations
The rapid growth of biomedical knowledge has made the validation of automatically generated biological annotations a major bottleneck in biomedical curation. While computational methods can rapidly produce large numbers …
Knowledge GraphsLeveraging Multi-Rater Annotations to Calibrate Object Detectors in Microscopy Imaging
Deep learning-based object detectors have achieved impressive performance in microscopy imaging, yet their confidence estimates often lack calibration, limiting their reliability for biomedical applications. In this work…
BIOSSES: A Semantic Sentence Similarity Estimation System for the Biomedical Domain
Motivation: The amount of information available in textual format is rapidly increasing in the biomedical domain. Therefore, natural language processing (NLP) applications are becoming increasingly important to facilitat…
RetrievalSemantic SimilaritySemantic Textual SimilaritySentence+3