A probabilistic database approach to autoencoder-based data cleaning
Data quality problems are a large threat in data science. In this paper, we propose a data-cleaning autoencoder capable of near-automatic data quality improvement. It learns the structure and dependencies in the data and uses it as evidence to identify and correct doubtful values. We apply a probabilistic database approach to represent weak and strong evidence for attribute value repairs. A theoretical framework is provided, and experiments show that it can remove significant amounts of noise (i.e., data quality problems) from categorical and numeric probabilistic data. Our method does not require clean data. We do, however, show that manually cleaning a small fraction of the data significantly improves performance.
Code (1)
Tasks
AttributeSimilar Papers 제목 키워드 기반
PClean: Bayesian Data Cleaning at Scale with Domain-Specific Probabilistic Programming
Data cleaning is naturally framed as probabilistic inference in a generative model of ground-truth data and likely errors, but the diversity of real-world error patterns and the hardness of inference make Bayesian approa…
DiversityProbabilistic ProgrammingAn epistemic approach to model uncertainty in data-graphs
Graph databases are becoming widely successful as data models that allow to effectively represent and process complex relationships among various types of data. As with any other type of data repository, graph databases …
BayesDB: A probabilistic programming system for querying the probable implications of data
Is it possible to make statistical inference broadly accessible to non-statisticians without sacrificing mathematical rigor or inference quality? This paper describes BayesDB, a probabilistic programming platform that ai…
Probabilistic ProgrammingCURED: Creating, Understanding, and Repairing Errors Demonstrator
Detecting and cleaning errors in tabular data is a prerequisite for data intense software applications. Recent research at the intersection of Machine Learning (ML) and Database Management Systems (DBMS) highlights the p…
BClean: A Bayesian Data Cleaning System
There is a considerable body of work on data cleaning which employs various principles to rectify erroneous data and transform a dirty dataset into a cleaner one. One of prevalent approaches is probabilistic methods, inc…
Bayesian Inferencegraph partitioning