On Performance Estimation in Automatic Algorithm Configuration
Over the last decade, research on automated parameter tuning, often referred to as automatic algorithm configuration (AAC), has made significant progress. Although the usefulness of such tools has been widely recognized in real world applications, the theoretical foundations of AAC are still very weak. This paper addresses this gap by studying the performance estimation problem in AAC. More specifically, this paper first proves the universal best performance estimator in a practical setting, and then establishes theoretical bounds on the estimation error, i.e., the difference between the training performance and the true performance for a parameter configuration, considering finite and infinite configuration spaces respectively. These findings were verified in extensive experiments conducted on four algorithm configuration scenarios involving different problem domains. Moreover, insights for enhancing existing AAC methods are also identified.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Discovering Multiple Algorithm Configurations
Many practitioners in robotics regularly depend on classic, hand-designed algorithms. Often the performance of these algorithms is tuned across a dataset of annotated examples which represent typical deployment condition…
Depth EstimationMotion PlanningVisual OdometryParaPose: Parameter and Domain Randomization Optimization for Pose Estimation using Synthetic Data
Pose estimation is the task of determining the 6D position of an object in a scene. Pose estimation aid the abilities and flexibility of robotic set-ups. However, the system must be configured towards the use case to per…
Pose EstimationDetection of 3D Bounding Boxes of Vehicles Using Perspective Transformation for Accurate Speed Measurement
Detection and tracking of vehicles captured by traffic surveillance cameras is a key component of intelligent transportation systems. We present an improved version of our algorithm for detection of 3D bounding boxes of …
Vehicle Speed EstimationParamILS: An Automatic Algorithm Configuration Framework
The identification of performance-optimizing parameter settings is an important part of the development and application of algorithms. We describe an automatic framework for this algorithm configuration problem. More for…
Hyperparameter OptimizationOnline unsupervised deep unfolding for MIMO channel estimation
Channel estimation is a difficult problem in MIMO systems. Using a physical model allows to ease the problem, injecting a priori information based on the physics of propagation. However, such models rest on simplifying a…
Efficient Neural Network