Time-Fair Benchmarking for Metaheuristics: A Restart-Fair Protocol for Fixed-Time Comparisons
Numerous purportedly improved metaheuristics claim superior performance based on equivalent function evaluations (FEs), yet often conceal additional computational burdens in more intensive iterations, preprocessing stages, or hyperparameter tuning. This paper posits that wall-clock time, rather than solely FEs, should serve as the principal budgetary constraint for equitable comparisons. We formalize a fixed-time, restart-fair benchmarking protocol wherein each algorithm is allotted an identical wall-clock time budget per problem instance, permitting unrestricted utilization of restarts, early termination criteria, and internal adaptive mechanisms. We advocate for the adoption of anytime performance curves, expected running time (ERT) metrics, and performance profiles that employ time as the cost measure, all aimed at predefined targets. Furthermore, we introduce a concise, reproducible checklist to standardize reporting practices and mitigate undisclosed computational overheads. This approach fosters more credible and practically relevant evaluations of metaheuristic algorithms.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Fairness-aware PageRank via Edge Reweighting
Link-analysis algorithms, such as PageRank, are instrumental in understanding the structural dynamics of networks by evaluating the importance of individual vertices based on their connectivity. Recently, with the rising…
FairX: A comprehensive benchmarking tool for model analysis using fairness, utility, and explainability
We present FairX, an open-source Python-based benchmarking tool designed for the comprehensive analysis of models under the umbrella of fairness, utility, and eXplainability (XAI). FairX enables users to train benchmarki…
BenchmarkingFairnessFFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methods
This paper introduces the Fair Fairness Benchmark (\textsf{FFB}), a benchmarking framework for in-processing group fairness methods. Ensuring fairness in machine learning is important for ethical compliance. However, the…
BenchmarkingFairnessGPURevisiting Pre-processing Group Fairness: A Modular Benchmarking Framework
As machine learning systems become increasingly integrated into high-stakes decision-making processes, ensuring fairness in algorithmic outcomes has become a critical concern. Methods to mitigate bias typically fall into…
Ensuring User-side Fairness in Dynamic Recommender Systems
User-side group fairness is crucial for modern recommender systems, aiming to alleviate performance disparities among user groups defined by sensitive attributes like gender, race, or age. In the ever-evolving landscape …
FairnessRecommendation SystemsRe-Ranking