A different perspective on a scale for pairwise comparisons
One of the major challenges for collective intelligence is inconsistency, which is unavoidable whenever subjective assessments are involved. Pairwise comparisons allow one to represent such subjective assessments and to process them by analyzing, quantifying and identifying the inconsistencies. We propose using smaller scales for pairwise comparisons and provide mathematical and practical justifications for this change. Our postulate's aim is to initiate a paradigm shift in the search for a better scale construction for pairwise comparisons. Beyond pairwise comparisons, the results presented may be relevant to other methods using subjective scales. Keywords: pairwise comparisons, collective intelligence, scale, subjective assessment, inaccuracy, inconsistency.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Decreasing Annotation Burden of Pairwise Comparisons with Human-in-the-Loop Sorting: Application in Medical Image Artifact Rating
Ranking by pairwise comparisons has shown improved reliability over ordinal classification. However, as the annotations of pairwise comparisons scale quadratically, this becomes less practical when the dataset is large. …
Ordinal ClassificationPrediction-Powered Ranking of Large Language Models
Large language models are often ranked according to their level of alignment with human preferences -- a model is better than other models if its outputs are more frequently preferred by humans. One of the popular ways t…
ChatbotLanguage ModellingLarge Language ModelPredictionKnockout LLM Assessment: Using Large Language Models for Evaluations through Iterative Pairwise Comparisons
Large Language Models (LLMs) have shown to be effective evaluators across various domains such as machine translations or the scientific domain. Current LLM-as-a-Judge approaches rely mostly on individual assessments or …
Machine TranslationAggregation of pairwise comparisons with reduction of biases
We study the problem of ranking from crowdsourced pairwise comparisons. Answers to pairwise tasks are known to be affected by the position of items on the screen, however, previous models for aggregation of pairwise comp…
PositionPreference learning along multiple criteria: A game-theoretic perspective
The literature on ranking from ordinal data is vast, and there are several ways to aggregate overall preferences from pairwise comparisons between objects. In particular, it is well known that any Nash equilibrium of the…
Autonomous Driving