Wizard of Errors: Introducing and Evaluating Machine Learning Errors in Wizard of Oz Studies
When designing Machine Learning (ML) enabled solutions, designers often need to simulate ML behavior through the Wizard of Oz (WoZ) approach to test the user experience before the ML model is available. Although reproducing ML errors is essential for having a good representation, they are rarely considered. We introduce Wizard of Errors (WoE), a tool for conducting WoZ studies on ML-enabled solutions that allows simulating ML errors during user experience assessment. We explored how this system can be used to simulate the behavior of a computer vision model. We tested WoE with design students to determine the importance of considering ML errors in design, the relevance of using descriptive error types instead of confusion matrix, and the suitability of manual error control in WoZ studies. Our work identifies several challenges, which prevent realistic error representation by designers in such studies. We discuss the implications of these findings for design.
Code (0)
등록된 구현이 없습니다.
Tasks
DescriptiveMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Improvement on Extrapolation of Species Abundance Distribution Across Scales from Moments Across Scales
Raw moments are used as a way to estimate species abundance distribution. The almost linear pattern of the log transformation of raw moments across scales allow us to extrapolate species abundance distribution for larger…
Cipher: A Prototype Game-with-a-Purpose for Detecting Errors in Text
Errors commonly exist in machine-generated documents and publication materials; however, some correction algorithms do not perform well for complex errors and it is costly to employ humans to do the task. To solve the pr…
The PHOTON Wizard -- Towards Educational Machine Learning Code Generators
Despite the tremendous efforts to democratize machine learning, especially in applied-science, the application is still often hampered by the lack of coding skills. As we consider programmatic understanding key to buildi…
BIG-bench Machine LearningvalidDaLA: Danish Linguistic Acceptability Evaluation Guided by Real World Errors
We present an enhanced benchmark for evaluating linguistic acceptability in Danish. We first analyze the most common errors found in written Danish. Based on this analysis, we introduce a set of fourteen corruption funct…
Linguistic AcceptabilityMISMATCH: Fine-grained Evaluation of Machine-generated Text with Mismatch Error Types
With the growing interest in large language models, the need for evaluating the quality of machine text compared to reference (typically human-generated) text has become focal attention. Most recent works focus either on…
Sentence