AI-Compass: A Comprehensive and Effective Multi-module Testing Tool for AI Systems
AI systems, in particular with deep learning techniques, have demonstrated superior performance for various real-world applications. Given the need for tailored optimization in specific scenarios, as well as the concerns related to the exploits of subsurface vulnerabilities, a more comprehensive and in-depth testing AI system becomes a pivotal topic. We have seen the emergence of testing tools in real-world applications that aim to expand testing capabilities. However, they often concentrate on ad-hoc tasks, rendering them unsuitable for simultaneously testing multiple aspects or components. Furthermore, trustworthiness issues arising from adversarial attacks and the challenge of interpreting deep learning models pose new challenges for developing more comprehensive and in-depth AI system testing tools. In this study, we design and implement a testing tool, \tool, to comprehensively and effectively evaluate AI systems. The tool extensively assesses multiple measurements towards adversarial robustness, model interpretability, and performs neuron analysis. The feasibility of the proposed testing tool is thoroughly validated across various modalities, including image classification, object detection, and text classification. Extensive experiments demonstrate that \tool is the state-of-the-art tool for a comprehensive assessment of the robustness and trustworthiness of AI systems. Our research sheds light on a general solution for AI systems testing landscape.
Code (0)
등록된 구현이 없습니다.
Tasks
Adversarial Robustnessimage-classificationImage Classificationobject-detectionObject Detectiontext-classificationText ClassificationSimilar Papers 제목 키워드 기반
Assessing behaviour coverage in a multi-agent system simulation for autonomous vehicle testing
As autonomous vehicle technology advances, ensuring the safety and reliability of these systems becomes paramount. Consequently, comprehensive testing methodologies are essential to evaluate the performance of autonomous…
Autonomous VehiclesAutonomous DrivingCompassDock: Comprehensive Accurate Assessment Approach for Deep Learning-Based Molecular Docking in Inference and Fine-Tuning
Datasets used for molecular docking, such as PDBBind, contain technical variability - they are noisy. Although the origins of the noise have been discussed, a comprehensive analysis of the physical, chemical, and bioacti…
Molecular DockingBenchmarking and Improving Compositional Generalization of Multi-aspect Controllable Text Generation
Compositional generalization, representing the model's ability to generate text with new attribute combinations obtained by recombining single attributes from the training data, is a crucial property for multi-aspect con…
AttributeBenchmarkingMeta-LearningText GenerationFPGA-Based Material Testing Machine Controller
In the realm of contemporary materials testing, the demand for scalability, adaptability, parallelism, and speed has surged due to the proliferation of diverse materials and testing standards. Traditional controller-base…
Effective Black Box Testing of Sentiment Analysis Classification Networks
Transformer-based neural networks have demonstrated remarkable performance in natural language processing tasks such as sentiment analysis. Nevertheless, the issue of ensuring the dependability of these complicated archi…
Sentiment Analysis