Litescale: A Lightweight Tool for Best-worst Scaling Annotation
Best-worst Scaling (BWS) is a methodology for annotation based on comparing and ranking instances, rather than classifying or scoring individual instances. Studies have shown the efficacy of this methodology applied to NLP tasks in terms of a higher quality of the datasets produced by following it. In this system demonstration paper, we present Litescale, a free software library to create and manage BWS annotation tasks. Litescale computes the tuples to annotate, manages the users and the annotation process, and creates the final gold standard. The functionalities of Litescale can be accessed programmatically through a Python module, or via two alternative user interfaces, a textual console-based one and a graphical Web-based one. We further developed and deployed a fully online version of Litescale complete with multi-user support.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Capturing Reliable Fine-Grained Sentiment Associations by Crowdsourcing and Best--Worst Scaling
"You are an expert annotator": Automatic Best-Worst-Scaling Annotations for Emotion Intensity Modeling
Labeling corpora constitutes a bottleneck to create models for new tasks or domains. Large language models mitigate the issue with automatic corpus labeling methods, particularly for categorical annotations. Some NLP tas…
Language ModelingLanguage ModellingLarge Language ModelregressionInvestigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries
Current pre-trained models applied for summarization are prone to factual inconsistencies which misrepresent the source text. Thus, evaluating the factual consistency of summaries is necessary to develop better models. H…
Baby Bear: Seeking a Just Right Rating Scale for Scalar Annotations
Our goal is a mechanism for efficiently assigning scalar ratings to each of a large set of elements. For example, "what percent positive or negative is this product review?" When sample sizes are small, prior work has ad…
Learning-To-RankBest-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation
Rating scales are a widely used method for data annotation; however, they present several challenges, such as difficulty in maintaining inter- and intra-annotator consistency. Best-worst scaling (BWS) is an alternative m…