Absolute Ranking: An Essential Normalization for Benchmarking Optimization Algorithms
Evaluating performance across optimization algorithms on many problems presents a complex challenge due to the diversity of numerical scales involved. Traditional data processing methods, such as hypothesis testing and Bayesian inference, often employ ranking-based methods to normalize performance values across these varying scales. However, a significant issue emerges with this ranking-based approach: the introduction of new algorithms can potentially disrupt the original rankings. This paper extensively explores the problem, making a compelling case to underscore the issue and conducting a thorough analysis of its root causes. These efforts pave the way for a comprehensive examination of potential solutions. Building on this research, this paper introduces a new mathematical model called "absolute ranking" and a sampling-based computational method. These contributions come with practical implementation recommendations, aimed at providing a more robust framework for addressing the challenge of numerical scale variation in the assessment of performance across multiple algorithms and problems.
Code (0)
등록된 구현이 없습니다.
Tasks
Bayesian InferenceBenchmarkingDiversitySimilar Papers 제목 키워드 기반
Pareto-Optimal Anytime Algorithms via Bayesian Racing
Selecting an optimization algorithm requires comparing candidates across problem instances, but the computational budget for deployment is often unknown at benchmarking time. Current methods either collapse anytime perfo…
Bayesian InferenceComprehensive Machine Learning Benchmarking for Fringe Projection Profilometry with Photorealistic Synthetic Data
Machine learning approaches for fringe projection profilometry (FPP) are hindered by the lack of large, diverse datasets and standardized benchmarking protocols. This paper introduces the first open-source, photorealisti…
Improved Stock Price Movement Classification Using News Articles Based on Embeddings and Label Smoothing
Stock price movement prediction is a challenging and essential problem in finance. While it is well established in modern behavioral finance that the share prices of related stocks often move after the release of news vi…
ArticlesSmooth Operator: Smooth Verifiable Reward Activates Spatial Reasoning Ability of Vision-Language Model
Vision-Language Models (VLMs) face a critical bottleneck in achieving precise numerical prediction for 3D scene understanding. Traditional reinforcement learning (RL) approaches, primarily based on relative ranking, ofte…
Reinforcement LearningScene UnderstandingSpatial ReasoningEfficient Benchmarking of AI Agents
Evaluating AI agents on comprehensive benchmarks is expensive because each evaluation requires interactive rollouts with tool use and multi-step reasoning. We study whether small task subsets can preserve agent rankings …