Aligning Intraobserver Agreement by Transitivity
Annotation reproducibility and accuracy rely on good consistency within annotators. We propose a novel method for measuring within annotator consistency or annotator Intraobserver Agreement (IA). The proposed approach is based on transitivity, a measure that has been thoroughly studied in the context of rational decision-making. The transitivity measure, in contrast with the commonly used test-retest strategy for annotator IA, is less sensitive to the several types of bias introduced by the test-retest strategy. We present a representation theorem to the effect that relative judgement data that meet transitivity can be mapped to a scale (in terms of measurement theory). We also discuss a further application of transitivity as part of data collection design for addressing the problem of the quadratic complexity of data collection of relative judgements.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingSimilar Papers 제목 키워드 기반
ELSPR: Evaluator LLM Training Data Self-Purification on Non-Transitive Preferences via Tournament Graph Reconstruction
Large language models (LLMs) are widely used as evaluators for open-ended tasks, while previous research has emphasized biases in LLM evaluations, the issue of non-transitivity in pairwise comparisons remains unresolved:…
Graph ReconstructionAutomated title and abstract screening for scoping reviews using the GPT-4 Large Language Model
Scoping reviews, a type of literature review, require intensive human effort to screen large numbers of scholarly sources for their relevance to the review objectives. This manuscript introduces GPTscreenR, a package for…
Language ModelingLanguage ModellingLarge Language ModelSpecificityA counter example to the theorems of social preference transitivity and social choice set existence under the majority rule
I present an example in which the individuals' preferences are strict orderings, and under the majority rule, a transitive social ordering can be obtained and thus a non-empty choice set can also be obtained. However, th…
What Should/Do/Can LSTMs Learn When Parsing Auxiliary Verb Constructions?
There is a growing interest in investigating what neural NLP models learn about language. A prominent open question is the question of whether or not it is necessary to model hierarchical structure. We present a linguist…
Dependency ParsingDiagnosticOpen-Ended Question AnsweringTesting the Annotation Consistency of Hallidayan Transitivity Processes: A Multi-variable Structural Approach
SFL seeks to explain identifiable, observable phenomena of language use in context through the application of a theoretical framework which models language as a functional, meaning making system (Halliday & Matthiessen 2…