Efficient Online Scalar Annotation with Bounded Support
We describe a novel method for efficiently eliciting scalar annotations for dataset construction and system quality estimation by human judgments. We contrast direct assessment (annotators assign scores to items directly), online pairwise ranking aggregation (scores derive from annotator comparison of items), and a hybrid approach (EASL: Efficient Annotation of Scalar Labels) proposed here. Our proposal leads to increased correlation with ground truth, at far greater annotator efficiency, suggesting this strategy as an improved mechanism for dataset creation and manual system evaluation.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Application of a Novel Model Reduction Technique to the Assessment of Boundedness/Stability of Some Delay Time-Varying Vector Nonlinear Systems
Assessing the boundedness and stability of vector nonlinear systems with variable delays and coefficients remains a challenging problem with broad applications in science and engineering. Existing methods tend to produce…
Instrumental Variables based DREM for Online Asymptotic Identification of Perturbed Linear Systems
Existing online continuous-time parameter estimation laws provide exact (asymptotic/exponential or finite/fixed time) identification of dynamical linear/nonlinear systems parameters only if the external perturbations are…
parameter estimationRiccati updates for online linear quadratic control
We study an online setting of the linear quadratic Gaussian optimal control problem on a sequence of cost functions, where similar to classical online optimization, the future decisions are made by only knowing the cost …
On the global convergence of gradient descent for wide shallow models with bounded nonlinearities
A surprising phenomenon in the training of neural networks is the ability of gradient descent to find global minimizers of the training loss despite its non-convexity. Following earlier works, we investigate this behavio…
Scalar Field Mapping with Adaptive High-Intensity Region Avoidance
This research is motivated by a scenario where a group of UAVs is assigned to map an unknown scalar field, with the imperative of maintaining a safe distance from the sources of the field to evade detection or damage. Th…