QUORUM: QUality-Optimized Routing Using Multiple annotators
Data annotation remains a central bottleneck in natural language processing, requiring human effort to obtain high-quality labels at scale. While Large Language Models (LLMs) offer a fast and cost-effective alternative, their reliability is highly instance-dependent: they perform well on simple inputs but often fail on examples requiring nuanced reasoning or contextual understanding. In this work, we address this challenge with QUORUM (QUality-Optimized Routing Using Multiple annotators), a budget-aware routing framework that dynamically assigns each instance to human or LLM annotators under a fixed annotation budget. Unlike prior approaches relying on model confidence or uncertainty estimates, QUORUM leverages feature-based signals to estimate instance difficulty and supports multiple annotations per instance, combining them through agreement-based rewards to improve reliability. We evaluate QUORUM across diverse closed- and open-ended annotation tasks in English and multilingual settings, and QUORUM improves annotation quality by up to 34.4% while reducing costs by 8.8% over competing methods. Code can be found at https://github.com/amazon-science/QUORUM.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
The Discrete Charm of the MLP: Binary Routing of Continuous Signals in Transformer Feed-Forward Layers
We show that MLP layers in transformer language models perform binary routing of continuous signals: the decision of whether a token needs nonlinear processing is well-captured by binary neuron activations, even though t…
SQUAD: Scalable Quorum Adaptive Decisions via ensemble of early exit neural networks
Early-exit neural networks have become popular for reducing inference latency by allowing intermediate predictions when sufficient confidence is achieved. However, standard approaches typically rely on single-model confi…
Neural Architecture SearchEnsemble LearningRobust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints
We study the problem of routing queries to large language models (LLMs) under cost, GPU resources, and concurrency constraints. Prior per-query routing methods often fail to control batch-level cost, especially under non…
Estimating the Level of Dialectness Predicts Interannotator Agreement in Multi-dialect Arabic Datasets
On annotating multi-dialect Arabic datasets, it is common to randomly assign the samples across a pool of native Arabic speakers. Recent analyses recommended routing dialectal samples to native speakers of their respecti…
SentenceSentence ClassificationAutocrine signaling and quorum sensing: Extreme ends of a common spectrum
"Secrete-and-sense cells" can communicate by secreting a signaling molecule while also producing a receptor that detects the molecule. The cell can potentially "talk" to itself ("self-communication") or talk to neighbori…