Survive the Schema Changes: Integration of Unmanaged Data Using Deep Learning
Data is the king in the age of AI. However data integration is often a laborious task that is hard to automate. Schema change is one significant obstacle to the automation of the end-to-end data integration process. Although there exist mechanisms such as query discovery and schema modification language to handle the problem, these approaches can only work with the assumption that the schema is maintained by a database. However, we observe diversified schema changes in heterogeneous data and open data, most of which has no schema defined. In this work, we propose to use deep learning to automatically deal with schema changes through a super cell representation and automatic injection of perturbations to the training data to make the model robust to schema changes. Our experimental results demonstrate that our proposed approach is effective for two real-world data integration scenarios: coronavirus data integration, and machine log integration.
Code (0)
등록된 구현이 없습니다.
Tasks
Data IntegrationDeep LearningSimilar Papers 제목 키워드 기반
Compound Schema Registry
Schema evolution is critical in managing database systems to ensure compatibility across different data versions. A schema registry typically addresses the challenges of schema evolution in real-time data streaming by ma…
ATEAM: Knowledge Integration from Federated Datasets for Vehicle Feature Extraction using Annotation Team of Experts
The vehicle recognition area, including vehicle make-model recognition (VMMR), re-id, tracking, and parts-detection, has made significant progress in recent years, driven by several large-scale datasets for each task. Th…
Fiber Nonlinearity Mitigation via the Parzen Window Classifier for Dispersion Managed and Unmanaged Links
Machine learning techniques have recently received significant attention as promising approaches to deal with the optical channel impairments, and in particular, the nonlinear effects. In this work, a machine learning-ba…
BIG-bench Machine LearningA Machine Learning-Based Detection Technique for Optical Fiber Nonlinearity Mitigation
We investigate the performance of a machine learning classification technique, called the Parzen window, to mitigate the fiber nonlinearity in the context of dispersion managed and dispersion unmanaged systems. The techn…
BIG-bench Machine LearningGeneral ClassificationA Closed-Form Approximation of the Gaussian Noise Model in the Presence of Inter-Channel Stimulated Raman Scattering
An accurate, closed-form expression evaluating the nonlinear interference (NLI) power in coherent optical transmission systems in the presence of inter-channel stimulated Raman scattering (ISRS) is derived. The analytica…
FormNumerical Integration