Ranking and benchmarking framework for sampling algorithms on synthetic data streams
In the fields of big data, AI, and streaming processing, we work with large amounts of data from multiple sources. Due to memory and network limitations, we process data streams on distributed systems to alleviate computational and network loads. When data streams with non-uniform distributions are processed, we often observe overloaded partitions due to the use of simple hash partitioning. To tackle this imbalance, we can use dynamic partitioning algorithms that require a sampling algorithm to precisely estimate the underlying distribution of the data stream. There is no standardized way to test these algorithms. We offer an extensible ranking framework with benchmark and hyperparameter optimization capabilities and supply our framework with a data generator that can handle concept drifts. Our work includes a generator for dynamic micro-bursts that we can apply to any data stream. We provide algorithms that react to concept drifts and compare those against the state-of-the-art algorithms using our framework.
Code (1)
Tasks
BenchmarkingHyperparameter OptimizationSimilar Papers 제목 키워드 기반
Is Synthetic Dataset Reliable for Benchmarking Generalizable Person Re-Identification?
Recent studies show that models trained on synthetic datasets are able to achieve better generalizable person re-identification (GPReID) performance than that trained on public real-world datasets. On the other hand, due…
BenchmarkingGeneralizable Person Re-identificationPerson Re-IdentificationAbsolute Ranking: An Essential Normalization for Benchmarking Optimization Algorithms
Evaluating performance across optimization algorithms on many problems presents a complex challenge due to the diversity of numerical scales involved. Traditional data processing methods, such as hypothesis testing and B…
Bayesian InferenceBenchmarkingDiversityMO-IOHinspector: Anytime Benchmarking of Multi-Objective Algorithms using IOHprofiler
Benchmarking is one of the key ways in which we can gain insight into the strengths and weaknesses of optimization algorithms. In sampling-based optimization, considering the anytime behavior of an algorithm can provide …
BenchmarkingExperimental Designfseval: A Benchmarking Framework for Feature Selection and Feature Ranking Algorithms
The fseval Python package allows benchmarking Feature Selection and Feature Ranking algorithms on a large scale, and facilitates the comparison of multiple algorithms in a systematic way. In particular, fseval enables us…
Automated Feature EngineeringBenchmarkingClassification with Costly FeaturesDistributed Computing+3A multi-schematic classifier-independent oversampling approach for imbalanced datasets
Over 85 oversampling algorithms, mostly extensions of the SMOTE algorithm, have been built over the past two decades, to solve the problem of imbalanced datasets. However, it has been evident from previous studies that d…
Benchmarking