Divide-and-conquer methods for big data analysis
In the context of big data analysis, the divide-and-conquer methodology refers to a multiple-step process: first splitting a data set into several smaller ones; then analyzing each set separately; finally combining results from each analysis together. This approach is effective in handling large data sets that are unsuitable to be analyzed entirely by a single computer due to limits either from memory storage or computational time. The combined results will provide a statistical inference which is similar to the one from analyzing the entire data set. This article reviews some recently developments of divide-and-conquer methods in a variety of settings, including combining based on parametric, semiparametric and nonparametric models, online sequential updating methods, among others. Theoretical development on the efficiency of the divide-and-conquer methods is also discussed.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Divide and Conquer in Video Anomaly Detection: A Comprehensive Review and New Approach
Video anomaly detection is a complex task, and the principle of "divide and conquer" is often regarded as an effective approach to tackling intricate issues. It's noteworthy that recent methods in video anomaly detection…
Anomaly DetectionPhilosophyVideo Anomaly DetectionTwo provably consistent divide and conquer clustering algorithms for large networks
In this article, we advance divide-and-conquer strategies for solving the community detection problem in networks. We propose two algorithms which perform clustering on a number of small subgraphs and finally patches the…
ClusteringCommunity DetectionVocal Bursts Valence PredictionDivide and Conquer Local Average Regression
The divide and conquer strategy, which breaks a massive data set into a se- ries of manageable data blocks, and then combines the independent results of data blocks to obtain a final decision, has been recognized as a st…
regressionOptimal Tuning for Divide-and-conquer Kernel Ridge Regression with Massive Data
Divide-and-conquer is a powerful approach for large and massive data analysis. In the nonparameteric regression setting, although various theoretical frameworks have been established to achieve optimality in estimat…
regressionTwo-sample testingStatistical Optimality of Divide and Conquer Kernel-based Functional Linear Regression
Previous analysis of regularized functional linear regression in a reproducing kernel Hilbert space (RKHS) typically requires the target function to be contained in this kernel space. This paper studies the convergence p…
regression