How Much Automation Does a Data Scientist Want?
Data science and machine learning (DS/ML) are at the heart of the recent advancements of many Artificial Intelligence (AI) applications. There is an active research thread in AI, \autoai, that aims to develop systems for automating end-to-end the DS/ML Lifecycle. However, do DS and ML workers really want to automate their DS/ML workflow? To answer this question, we first synthesize a human-centered AutoML framework with 6 User Role/Personas, 10 Stages and 43 Sub-Tasks, 5 Levels of Automation, and 5 Types of Explanation, through reviewing research literature and marketing reports. Secondly, we use the framework to guide the design of an online survey study with 217 DS/ML workers who had varying degrees of experience, and different user roles "matching" to our 6 roles/personas. We found that different user personas participated in distinct stages of the lifecycle -- but not all stages. Their desired levels of automation and types of explanation for AutoML also varied significantly depending on the DS/ML stage and the user persona. Based on the survey results, we argue there is no rationale from user needs for complete automation of the end-to-end DS/ML lifecycle. We propose new next steps for user-controlled DS/ML automation.
Code (0)
등록된 구현이 없습니다.
Tasks
AutoMLMarketingSurveySimilar Papers 제목 키워드 기반
When Performance is not Enough -- A Multidisciplinary View on Clinical Decision Support
Scientific publications about machine learning in healthcare are often about implementing novel methods and boosting the performance - at least from a computer science perspective. However, beyond such often short-lived …
BIG-bench Machine LearningPrediction Factory: automated development and collaborative evaluation of predictive models
In this paper, we present a data science automation system called Prediction Factory. The system uses several key automation algorithms to enable data scientists to rapidly develop predictive models and share them with d…
Visualizing Weights
The problem of understanding a neural network is a little bit like reverse engineering a large compiled binary of a computer program. In this analogy, the weights of the neural network are the compiled assembly instructi…
Automating Care by Self-maintainability for Full Laboratory Automation
The automation of experiments in life sciences and chemistry has significantly advanced with the development of various instruments and AI technologies. However, achieving full laboratory automation, where experiments co…
SchedulingA Vision for Semantically Enriched Data Science
The recent efforts in automation of machine learning or data science has achieved success in various tasks such as hyper-parameter optimization or model selection. However, key areas such as utilizing domain knowledge an…
Common Sense ReasoningData AugmentationModel Selection