Multilevel Stochastic Optimization for Imputation in Massive Medical Data Records
It has long been a recognized problem that many datasets contain significant levels of missing numerical data. A potentially critical predicate for application of machine learning methods to datasets involves addressing this problem. However, this is a challenging task. In this paper, we apply a recently developed multi-level stochastic optimization approach to the problem of imputation in massive medical records. The approach is based on computational applied mathematics techniques and is highly accurate. In particular, for the Best Linear Unbiased Predictor (BLUP) this multi-level formulation is exact, and is significantly faster and more numerically stable. This permits practical application of Kriging methods to data imputation problems for massive datasets. We test this approach on data from the National Inpatient Sample (NIS) data records, Healthcare Cost and Utilization Project (HCUP), Agency for Healthcare Research and Quality. Numerical results show that the multi-level method significantly outperforms current approaches and is numerically robust. It has superior accuracy as compared with methods recommended in the recent report from HCUP. Benchmark tests show up to 75% reductions in error. Furthermore, the results are also superior to recent state of the art methods such as discriminative deep learning.
Code (0)
등록된 구현이 없습니다.
Tasks
ImputationStochastic OptimizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Fast Imbalanced Classification of Healthcare Data with Missing Values
In medical domain, data features often contain missing values. This can create serious bias in the predictive modeling. Typical standard data mining methods often produce poor performance measures. In this paper, we prop…
ClassificationGeneral Classificationimbalanced classificationImputation+2Multilevel Weighted Support Vector Machine for Classification on Healthcare Data with Missing Values
This work is motivated by the needs of predictive analytics on healthcare data as represented by Electronic Medical Records. Such data is invariably problematic: noisy, with missing entries, with imbalance in classes of …
ClassificationGeneral ClassificationImputationMissing Values+1Stochastic Functional Analysis and Multilevel Vector Field Anomaly Detection
Massive vector field datasets are common in multi-spectral optical and radar sensors, among many other emerging areas of application. In this paper we develop a novel stochastic functional (data) analysis approach for de…
Anomaly DetectionOn the Parallel Complexity of Multilevel Monte Carlo in Stochastic Gradient Descent
In the stochastic gradient descent (SGD) for sequential simulations such as the neural stochastic differential equations, the Multilevel Monte Carlo (MLMC) method is known to offer better theoretical computational comple…
Adapting tree-based multiple imputation methods for multi-level data? A simulation study
When data have a hierarchical structure, such as students nested within classrooms, ignoring dependencies between observations can compromise the validity of imputation procedures. Standard tree-based imputation methods …
Imputation