Perspectives on Incorporating Expert Feedback into Model Updates
Machine learning (ML) practitioners are increasingly tasked with developing models that are aligned with non-technical experts' values and goals. However, there has been insufficient consideration on how practitioners should translate domain expertise into ML updates. In this paper, we consider how to capture interactions between practitioners and experts systematically. We devise a taxonomy to match expert feedback types with practitioner updates. A practitioner may receive feedback from an expert at the observation- or domain-level, and convert this feedback into updates to the dataset, loss function, or parameter space. We review existing work from ML and human-computer interaction to describe this feedback-update taxonomy, and highlight the insufficient consideration given to incorporating feedback from non-technical experts. We end with a set of open questions that naturally arise from our proposed taxonomy and subsequent survey.
Code (0)
등록된 구현이 없습니다.
Tasks
modelSimilar Papers 제목 키워드 기반
Lensing Machines: Representing Perspective in Latent Variable Models
Many datasets represent a combination of different ways of looking at the same data that lead to different generalizations. For example, a corpus with examples generated by different people may be mixtures of many perspe…
Ship Detection in SAR Images with Human-in-the-Loop
Synthetic aperture radar (SAR) has been extensively utilized in maritime domains due to its all-weather, all-day monitoring capabilities, particularly exhibiting significant value in ship detection. In recent years, deep…
Active LearningSAR Ship DetectionFeedbackLogs: Recording and Incorporating Stakeholder Feedback into Machine Learning Pipelines
Even though machine learning (ML) pipelines affect an increasing array of stakeholders, there is little work on how input from stakeholders is recorded and incorporated. We propose FeedbackLogs, addenda to existing docum…
Anomaly detection with superexperts under delayed feedback
The increasing connectivity of data and cyber-physical systems has resulted in a growing number of cyber-attacks. Real-time detection of such attacks, through the identification of anomalous activity, is required so that…
Anomaly DetectionUnsupervised Anomaly DetectionHow Useful is Intermittent, Asynchronous Expert Feedback for Bayesian Optimization?
Bayesian optimization (BO) is an integral part of automated scientific discovery -- the so-called self-driving lab -- where human inputs are ideally minimal or at least non-blocking. However, scientists often have strong…
Bayesian OptimizationBlockingscientific discovery