HARE: a Flexible Highlighting Annotator for Ranking and Exploration
Exploration and analysis of potential data sources is a significant challenge in the application of NLP techniques to novel information domains. We describe HARE, a system for highlighting relevant information in document collections to support ranking and triage, which provides tools for post-processing and qualitative analysis for model development and tuning. We apply HARE to the use case of narrative descriptions of mobility information in clinical data, and demonstrate its utility in comparing candidate embedding features. We provide a web-based interface for annotation visualization and document ranking, with a modular backend to support interoperability with existing annotation tools. Our system is available online at https://github.com/OSU-slatelab/HARE.
Code (1)
Tasks
Document RankingSimilar Papers 제목 키워드 기반
Contrastive ESA: Human Evaluation of Multiple Translations at Once
Current human evaluation of machine translation typically assesses single outputs in isolation, a paradigm that suffers from high annotator noise and cost. We introduce Contrastive Error Span Annotation (cESA), a protoco…
Machine TranslationLearning Multiple Tasks in Parallel with a Shared Annotator
We introduce a new multi-task framework, in which $K$ online learners are sharing a single annotator with limited bandwidth. On each round, each of the $K$ learners receives an input, and makes a prediction about the lab…
Binary ClassificationDocument ClassificationGeneral ClassificationMulti-Armed Bandits+2FuocChuVIP123 at CoMeDi Shared Task: Disagreement Ranking with XLM-Roberta Sentence Embeddings and Deep Neural Regression
This paper presents results of our system for CoMeDi Shared Task, focusing on Subtask 2: Disagreement Ranking. Our system leverages sentence embeddings generated by the paraphrase-xlm-r-multilingual-v1 model, combined wi…
SentenceSentence EmbeddingsXLM-RSTABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems
Human evaluation remains the primary standard for assessing modern AI systems, yet annotator disagreement, bias, and variability make system rankings fragile under standard majority vote aggregation. Majority vote discar…
Active Learning++: Incorporating Annotator's Rationale using Local Model Explanation
We propose a new active learning (AL) framework, Active Learning++, which can utilize an annotator's labels as well as its rationale. Annotators can provide their rationale for choosing a label by ranking input features …
Active LearningFeature Importance