Debugging Machine Learning Pipelines
Machine learning tasks entail the use of complex computational pipelines to reach quantitative and qualitative conclusions. If some of the activities in a pipeline produce erroneous or uninformative outputs, the pipeline may fail or produce incorrect results. Inferring the root cause of failures and unexpected behavior is challenging, usually requiring much human thought, and is both time-consuming and error-prone. We propose a new approach that makes use of iteration and provenance to automatically infer the root causes and derive succinct explanations of failures. Through a detailed experimental evaluation, we assess the cost, precision, and recall of our approach compared to the state of the art. Our source code and experimental data will be available for reproducibility and enhancement.
Code (1)
Tasks
BIG-bench Machine LearningSimilar Papers 제목 키워드 기반
TimeGym: Debugging for Time Series Modeling in Python
We introduce the TimeGym Forecasting Debugging Toolkit, a Python library for testing and debugging time series forecasting pipelines. TimeGym simplifies the testing forecasting pipeline by providing generic tests for for…
test driven developmentTime SeriesTime Series AnalysisTime Series ForecastingData Debugging with Shapley Importance over End-to-End Machine Learning Pipelines
Developing modern machine learning (ML) applications is data-centric, of which one fundamental challenge is to understand the influence of data quality to ML training -- "Which training examples are 'guilty' in making th…
BIG-bench Machine LearningFairnessRAG Without the Lag: Interactive Debugging for Retrieval-Augmented Generation Pipelines
Retrieval-augmented generation (RAG) pipelines have become the de-facto approach for building AI assistants with access to external, domain-specific knowledge. Given a user query, RAG pipelines typically first retrieve (…
Language ModelingLanguage ModellingLarge Language ModelRAG+2Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
Unlike code completion, debugging requires localizing faults and applying targeted edits. We observe that frontier LLMs often regenerate correct but over-edited solutions during debugging. To evaluate how far LLMs are fr…
Code CompletionSemPipes -- Optimizable Semantic Data Operators for Tabular Machine Learning Pipelines
Real-world machine learning on tabular data relies on complex data preparation pipelines for prediction, data integration, augmentation, and debugging. Designing these pipelines requires substantial domain expertise and …