Boosting K-means for Big Data by Fusing Data Streaming with Global Optimization
K-means clustering is a cornerstone of data mining, but its efficiency deteriorates when confronted with massive datasets. To address this limitation, we propose a novel heuristic algorithm that leverages the Variable Neighborhood Search (VNS) metaheuristic to optimize K-means clustering for big data. Our approach is based on the sequential optimization of the partial objective function landscapes obtained by restricting the Minimum Sum-of-Squares Clustering (MSSC) formulation to random samples from the original big dataset. Within each landscape, systematically expanding neighborhoods of the currently best (incumbent) solution are explored by reinitializing all degenerate and a varying number of additional centroids. Extensive and rigorous experimentation on a large number of real-world datasets reveals that by transforming the traditional local search into a global one, our algorithm significantly enhances the accuracy and efficiency of K-means clustering in big data environments, becoming the new state of the art in the field.
Code (0)
등록된 구현이 없습니다.
Tasks
Clusteringglobal-optimizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
SAKR: Enhancing Retrieval-Augmented Generation via Streaming Algorithm and K-Means Clustering
Retrieval-augmented generation (RAG) has achieved significant success in information retrieval to assist large language models LLMs because it builds an external knowledge database. However, it also has many problems, it…
Information RetrievalRAGRetrievalRetrieval-augmented GenerationCoresets for constrained k-median and k-means clustering in low dimensional Euclidean space
We study (Euclidean) $k$-median and $k$-means with constraints in the streaming model. There have been recent efforts to design unified algorithms to solve constrained $k$-means problems without using knowledge of the sp…
ClusteringStreaming k-means approximation
We provide a clustering algorithm that approximately optimizes the k-means objective, in the one-pass streaming setting. We make no assumptions about the data, and our algorithm is very light-weight in terms of memory, …
ClusteringKnowledge boosting during low-latency inference
Models for low-latency, streaming applications could benefit from the knowledge capacity of larger models, but edge devices cannot run these models due to resource constraints. A possible solution is to transfer hints du…
Speech SeparationStudy on the Assessment of the Quality of Experience of Streaming Video
Dynamic adaptive streaming over HTTP provides the work of most multimedia services, however, the nature of this technology further complicates the assessment of the QoE (Quality of Experience). In this paper, the influen…
regressionVideo Quality AssessmentVisual Question Answering (VQA)