Benchmarking Automatic Machine Learning Frameworks
AutoML serves as the bridge between varying levels of expertise when designing machine learning systems and expedites the data science process. A wide range of techniques is taken to address this, however there does not exist an objective comparison of these techniques. We present a benchmark of current open source AutoML solutions using open source datasets. We test auto-sklearn, TPOT, auto_ml, and H2O's AutoML solution against a compiled set of regression and classification datasets sourced from OpenML and find that auto-sklearn performs the best across classification datasets and TPOT performs the best across regression datasets.
Code (4)
Tasks
Automated Feature EngineeringAutoMLBenchmarkingBIG-bench Machine LearningClassificationGeneral ClassificationHyperparameter OptimizationregressionSimilar Papers 제목 키워드 기반
Benchmarking Contemporary Deep Learning Hardware and Frameworks:A Survey of Qualitative Metrics
This paper surveys benchmarking principles, machine learning devices including GPUs, FPGAs, and ASICs, and deep learning software frameworks. It also reviews these technologies with respect to benchmarking from the persp…
BenchmarkingBIG-bench Machine LearningDeep LearningMLonMCU: TinyML Benchmarking with Fast Retargeting
While there exist many ways to deploy machine learning models on microcontrollers, it is non-trivial to choose the optimal combination of frameworks and targets for a given application. Thus, automating the end-to-end be…
BenchmarkingBenchmarking the Linear Algebra Awareness of TensorFlow and PyTorch
Linear algebra operations, which are ubiquitous in machine learning, form major performance bottlenecks. The High-Performance Computing community invests significant effort in the development of architecture-specific opt…
BenchmarkingSelf-supervised Benchmark Lottery on ImageNet: Do Marginal Improvements Translate to Improvements on Similar Datasets?
Machine learning (ML) research strongly relies on benchmarks in order to determine the relative effectiveness of newly proposed models. Recently, a number of prominent research effort argued that a number of models that …
BenchmarkingSelf-Supervised LearningGNNBENCH: Fair and Productive Benchmarking for Single-GPU GNN System
We hypothesize that the absence of a standardized benchmark has allowed several fundamental pitfalls in GNN System design and evaluation that the community has overlooked. In this work, we propose GNNBench, a plug-and-pl…
BenchmarkingGPU