Parallel and Multi-Objective Falsification with Scenic and VerifAI
Falsification has emerged as an important tool for simulation-based verification of autonomous systems. In this paper, we present extensions to the Scenic scenario specification language and VerifAI toolkit that improve the scalability of sampling-based falsification methods by using parallelism and extend falsification to multi-objective specifications. We first present a parallelized framework that is interfaced with both the simulation and sampling capabilities of Scenic and the falsification capabilities of VerifAI, reducing the execution time bottleneck inherently present in simulation-based testing. We then present an extension of VerifAI's falsification algorithms to support multi-objective optimization during sampling, using the concept of rulebooks to specify a preference ordering over multiple metrics that can be used to guide the counterexample search process. Lastly, we evaluate the benefits of these extensions with a comprehensive set of benchmarks written in the Scenic language.
Code (1)
Similar Papers 제목 키워드 기반
Formal Analysis and Redesign of a Neural Network-Based Aircraft Taxiing System with VerifAI
We demonstrate a unified approach to rigorous design of safety-critical autonomous systems using the VerifAI toolkit for formal analysis of AI-based systems. VerifAI provides an integrated toolchain for tasks spanning th…
Probabilistic ProgrammingAddressing the IEEE AV Test Challenge with Scenic and VerifAI
This paper summarizes our formal approach to testing autonomous vehicles (AVs) in simulation for the IEEE AV Test Challenge. We demonstrate a systematic testing framework leveraging our previous work on formally-driven s…
Autonomous VehiclesProbabilistic ProgrammingVERIFAI: A Toolkit for the Design and Analysis of Artificial Intelligence-Based Systems
We present VERIFAI, a software toolkit for the formal design and analysis of systems that include artificial intelligence (AI) and machine learning (ML) components. VERIFAI particularly seeks to address challenges with a…
BIG-bench Machine LearningA Scenario-Based Platform for Testing Autonomous Vehicle Behavior Prediction Models in Simulation
Behavior prediction remains one of the most challenging tasks in the autonomous vehicle (AV) software stack. Forecasting the future trajectories of nearby agents plays a critical role in ensuring road safety, as it equip…
PredictionProbabilistic ProgrammingScenicRules: An Autonomous Driving Benchmark with Multi-Objective Specifications and Abstract Scenarios
Developing autonomous driving systems for complex traffic environments requires balancing multiple objectives, such as avoiding collisions, obeying traffic rules, and making efficient progress. In many situations, these …
Autonomous VehiclesAutonomous Driving