Model LineUpper: Supporting Interactive Model Comparison at Multiple Levels for AutoML
Automated Machine Learning (AutoML) is a rapidly growing set of technologies that automate the model development pipeline by searching model space and generating candidate models. A critical, final step of AutoML is human selection of a final model from dozens of candidates. In current AutoML systems, selection is supported only by performance metrics. Prior work has shown that in practice, people evaluate ML models based on additional criteria, such as the way a model makes predictions. Comparison may happen at multiple levels, from types of errors, to feature importance, to how the model makes predictions of specific instances. We developed \tool{} to support interactive model comparison for AutoML by integrating multiple Explainable AI (XAI) and visualization techniques. We conducted a user study in which we both evaluated the system and used it as a technology probe to understand how users perform model comparison in an AutoML system. We discuss design implications for utilizing XAI techniques for model comparison and supporting the unique needs of data scientists in comparing AutoML models.
Code (0)
등록된 구현이 없습니다.
Tasks
AutoMLExplainable Artificial Intelligence (XAI)Feature ImportancemodelSimilar Papers 제목 키워드 기반
TSExplorer: An interactive data annotation and exploration tool for time-series data
We present TSExplorer, a cross-platform tool for interactive annotation and exploration of time-series data. The tool enables users to inspect high-dimensional datasets through multiple complementary 2D visualizations de…
Evaluating Mixed-Initiative Procedural Level Design Tools using a Triple-Blind Mixed-Method User Study
Results from a triple-blind mixed-method user study into the effectiveness of mixed-initiative tools for the procedural generation of game levels are presented. A tool which generates levels using interactive evolutionar…
RISE: Interactive Visual Diagnosis of Fairness in Machine Learning Models
Evaluating fairness under domain shift is challenging because scalar metrics often obscure exactly where and how disparities arise. We introduce \textit{RISE} (Residual Inspection through Sorted Evaluation), an interacti…
Interactive Groupwise Comparison for Reinforcement Learning from Human Feedback
Reinforcement learning from human feedback (RLHF) has emerged as a key enabling technology for aligning AI behaviour with human preferences. The traditional way to collect data in RLHF is via pairwise comparisons: human …
Reinforcement LearningActive LearningFull State-Space Visualisation of the 8-Puzzle: Feasibility, Design, and Educational Use
Search algorithms are a foundational topic in artificial intelligence education, yet even simple domains can generate large state spaces that challenge learners' ability to form accurate mental models. This paper present…