Quantitative Projection Coverage for Testing ML-enabled Autonomous Systems
Systematically testing models learned from neural networks remains a crucial unsolved barrier to successfully justify safety for autonomous vehicles engineered using data-driven approach. We propose quantitative k-projection coverage as a metric to mediate combinatorial explosion while guiding the data sampling process. By assuming that domain experts propose largely independent environment conditions and by associating elements in each condition with weights, the product of these conditions forms scenarios, and one may interpret weights associated with each equivalence class as relative importance. Achieving full k-projection coverage requires that the data set, when being projected to the hyperplane formed by arbitrarily selected k-conditions, covers each class with number of data points no less than the associated weight. For the general case where scenario composition is constrained by rules, precisely computing k-projection coverage remains in NP. In terms of finding minimum test cases to achieve full coverage, we present theoretical complexity for important sub-cases and an encoding to 0-1 integer programming. We have implemented a research prototype that generates test cases for a visual object defection unit in automated driving, demonstrating the technological feasibility of our proposed coverage criterion.
Code (0)
등록된 구현이 없습니다.
Tasks
Autonomous VehiclesSimilar Papers 제목 키워드 기반
Assessing behaviour coverage in a multi-agent system simulation for autonomous vehicle testing
As autonomous vehicle technology advances, ensuring the safety and reliability of these systems becomes paramount. Consequently, comprehensive testing methodologies are essential to evaluate the performance of autonomous…
Autonomous VehiclesAutonomous DrivingProbabilistic Safety Verification for an Autonomous Ground Vehicle: A Situation Coverage Grid Approach
As industrial autonomous ground vehicles are increasingly deployed in safety-critical environments, ensuring their safe operation under diverse conditions is paramount. This paper presents a novel approach for their safe…
Stealthy Coverage Control for Human-enabled Real-Time 3D Reconstruction
In this paper, we propose a novel semi-autonomous image sampling strategy, called stealthy coverage control, for human-enabled 3D structure reconstruction. The present mission involves a fundamental problem: while the nu…
3D ReconstructionRL-STPA: Adapting System-Theoretic Hazard Analysis for Safety-Critical Reinforcement Learning
As reinforcement learning (RL) deployments expand into safety-critical domains, existing evaluation methods fail to systematically identify hazards arising from the black-box nature of neural network enabled policies and…
Reinforcement LearningDrone navigationMulti-Agent LLM Committees for Autonomous Software Beta Testing
Manual software beta testing is costly and time-consuming, while single-agent large language model (LLM) approaches suffer from hallucinations and inconsistent behavior. We propose a multi-agent committee framework in wh…