Leveraging Classification Metrics for Quantitative System-Level Analysis with Temporal Logic Specifications
In many autonomy applications, performance of perception algorithms is important for effective planning and control. In this paper, we introduce a framework for computing the probability of satisfaction of formal system specifications given a confusion matrix, a statistical average performance measure for multi-class classification. We define the probability of satisfaction of a linear temporal logic formula given a specific initial state of the agent and true state of the environment. Then, we present an algorithm to construct a Markov chain that represents the system behavior under the composition of the perception and control components such that the probability of the temporal logic formula computed over the Markov chain is consistent with the probability that the temporal logic formula is satisfied by our system. We illustrate this approach on a simple example of a car with pedestrian on the sidewalk environment, and compute the probability of satisfaction of safety requirements for varying parameters of the vehicle. We also illustrate how satisfaction probability changes with varied precision and recall derived from the confusion matrix. Based on our results, we identify several opportunities for future work in developing quantitative system-level analysis that incorporates perception models.
Code (0)
등록된 구현이 없습니다.
Tasks
Multi-class ClassificationSimilar Papers 제목 키워드 기반
Leveraging GANs for citation intent classification and its impact on citation network analysis
Citations play a fundamental role in the scientific ecosystem, serving as a foundation for tracking the flow of knowledge, acknowledging prior work, and assessing scholarly influence. In scientometrics, they are also cen…
Citation Intent Classificationintent-classificationIntent ClassificationIndependent Ethical Assessment of Text Classification Models: A Hate Speech Detection Case Study
An independent ethical assessment of an artificial intelligence system is an impartial examination of the system's development, deployment, and use in alignment with ethical values. System-level qualitative frameworks th…
counterfactualHate Speech Detectiontext-classificationText ClassificationMachine learning based biomedical image processing for echocardiographic images
The popularity of Artificial intelligence and machine learning have prompted researchers to use it in the recent researches. The proposed method uses K-Nearest Neighbor (KNN) algorithm for segmentation of medical images,…
regressionClassification-driven Single Image Dehazing
Most existing dehazing algorithms often use hand-crafted features or Convolutional Neural Networks (CNN)-based methods to generate clear images using pixel-level Mean Square Error (MSE) loss. The generated images general…
ClassificationGeneral Classificationimage-classificationImage Classification+2Color Image Segmentation Metrics
An automatic image segmentation procedure is an inevitable part of many image analyses and computer vision which deeply affect the rest of the system; therefore, a set of interactive segmentation evaluation methods can s…
Decision MakingImage SegmentationInteractive SegmentationSegmentation+1