Metrology for AI: From Benchmarks to Instruments
In this paper we present the first steps towards hardening the science of measuring AI systems, by adopting metrology, the science of measurement and its application, and applying it to human (crowd) powered evaluations. We begin with the intuitive observation that evaluating the performance of an AI system is a form of measurement. In all other science and engineering disciplines, the devices used to measure are called instruments, and all measurements are recorded with respect to the characteristics of the instruments used. One does not report mass, speed, or length, for example, of a studied object without disclosing the precision (measurement variance) and resolution (smallest detectable change) of the instrument used. It is extremely common in the AI literature to compare the performance of two systems by using a crowd-sourced dataset as an instrument, but failing to report if the performance difference lies within the capability of that instrument to measure. To illustrate the adoption of metrology to benchmark datasets we use the word similarity benchmark WS353 and several previously published experiments that use it for evaluation.
Code (1)
Tasks
Word SimilaritySimilar Papers 제목 키워드 기반
Model-based interfacing of large-scale metrology instruments
Metrology assisted assembly systems constitute cyber physical production systems relying on in-process sensor data as input to model-based control loops. These range from local, physical control loops, e.g. for robots to…
ManagementmodelBenchmarks as Microscopes: A Call for Model Metrology
Modern language models (LMs) pose a new challenge in capability assessment. Static benchmarks inevitably saturate without providing confidence in the deployment tolerances of LM-based systems, but developers nonetheless …
BenchmarkingmodelOptical and RF Metrology for 5G
Specification standards will soon be available for 5G mobile RF communications. What optical and electrical metrology is needed or available to support the development of the supporting optical communication systems? Dev…
Single-shot Adaptive Measurement for Quantum-enhanced Metrology
Quantum-enhanced metrology aims to estimate an unknown parameter such that the precision scales better than the shot-noise bound. Single-shot adaptive quantum-enhanced metrology (AQEM) is a promising approach that uses f…
Decision MakingReinforcement LearningBeyond Heisenberg Limit Quantum Metrology through Quantum Signal Processing
Leveraging quantum effects in metrology such as entanglement and coherence allows one to measure parameters with enhanced sensitivity. However, time-dependent noise can disrupt such Heisenberg-limited amplification. We p…