Distributed sequential method for analyzing massive data
To analyse a very large data set containing lengthy variables, we adopt a sequential estimation idea and propose a parallel divide-and-conquer method. We conduct several conventional sequential estimation procedures separately, and properly integrate their results while maintaining the desired statistical properties. Additionally, using a criterion from the statistical experiment design, we adopt an adaptive sample selection, together with an adaptive shrinkage estimation method, to simultaneously accelerate the estimation procedure and identify the effective variables. We confirm the cogency of our methods through theoretical justifications and numerical results derived from synthesized data sets. We then apply the proposed method to three real data sets, including those pertaining to appliance energy use and particulate matter concentration.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Asynchronous Training of Word Embeddings for Large Text Corpora
Word embeddings are a powerful approach for analyzing language and have been widely popular in numerous tasks in information retrieval and text mining. Training embeddings over huge corpora is computationally expensive b…
Information RetrievalRetrievalWord EmbeddingsFast Distributed k-Center Clustering with Outliers on Massive Data
Clustering large data is a fundamental problem with a vast number of applications. Due to the increasing size of data, practitioners interested in clustering have turned to distributed computation methods. In this work…
ClusteringDistributed ComputingCo-Located vs Distributed vs Semi-Distributed MIMO: Measurement-Based Evaluation
With the growing interest in cell-free massive multiple-input multiple-output (MIMO) systems, the benefits of single-antenna access points (APs) versus multi-antenna APs must be analyzed in order to optimize deployment. …
Distributed Dynamic Safe Screening Algorithms for Sparse Regularization
Distributed optimization has been widely used as one of the most efficient approaches for model training with massive samples. However, large-scale learning problems with both massive samples and high-dimensional feature…
Distributed OptimizationSequential In-Network Processing for Cell-Free Massive MIMO with Capacity-Constrained Parallel Radio Stripes
To ensure coherent signal processing across distributed Access Points (APs) in Cell-Free Massive Multiple-Input Multiple-Output (CF-mMIMO) systems, a fronthaul connection between the APs and a Central Processor (CP) is i…