Benchmarking in Optimization: Best Practice and Open Issues
This survey compiles ideas and recommendations from more than a dozen researchers with different backgrounds and from different institutes around the world. Promoting best practice in benchmarking is its main goal. The article discusses eight essential topics in benchmarking: clearly stated goals, well-specified problems, suitable algorithms, adequate performance measures, thoughtful analysis, effective and efficient designs, comprehensible presentations, and guaranteed reproducibility. The final goal is to provide well-accepted guidelines (rules) that might be useful for authors and reviewers. As benchmarking in optimization is an active and evolving field of research this manuscript is meant to co-evolve over time by means of periodic updates.
Code (0)
등록된 구현이 없습니다.
Tasks
BenchmarkingSimilar Papers 제목 키워드 기반
Metrics for Benchmarking and Uncertainty Quantification: Quality, Applicability, and a Path to Best Practices for Machine Learning in Chemistry
This review aims to draw attention to two issues of concern when we set out to make machine learning work in the chemical and materials domain, i.e., statistical loss function metrics for the validation and benchmarking …
BenchmarkingBIG-bench Machine LearningUncertainty QuantificationBenchmarking framework for machine learning classification from fNIRS data
Background: While efforts to establish best practices with functional near infrared spectroscopy (fNIRS) signal processing have been published, there are still no community standards for applying machine learning to fNIR…
BenchmarkingBrain Computer InterfaceClassificationTerraTorch: The Geospatial Foundation Models Toolkit
TerraTorch is a fine-tuning and benchmarking toolkit for Geospatial Foundation Models built on PyTorch Lightning and tailored for satellite, weather, and climate data. It integrates domain-specific data modules, pre-defi…
BenchmarkingDecoderEarth ObservationHyperparameter OptimizationThe Collective Knowledge project: making ML models more portable and reproducible with open APIs, reusable best practices and MLOps
This article provides an overview of the Collective Knowledge technology (CK or cKnowledge). CK attempts to make it easier to reproduce ML&systems research, deploy ML models in production, and adapt them to continuously …
Benchmarkingobject-detectionObject DetectionNeuroEvoBench: Benchmarking Evolutionary Optimizers for Deep Learning Applications
Recently, the Deep Learning community has become interested in evolutionary optimization (EO) as a means to address hard optimization problems, e.g. meta-learning through long inner loop unrolls or optimizing non-differe…
BenchmarkingDeep LearningMeta-Learning