Efficient Exploration of Image Classifier Failures with Bayesian Optimization and Text-to-Image Models
Image classifiers should be used with caution in the real world. Performance evaluated on a validation set may not reflect performance in the real world. In particular, classifiers may perform well for conditions that are frequently encountered during training, but poorly for other infrequent conditions. In this study, we hypothesize that recent advances in text-to-image generative models make them valuable for benchmarking computer vision models such as image classifiers: they can generate images conditioned by textual prompts that cause classifier failures, allowing failure conditions to be described with textual attributes. However, their generation cost becomes an issue when a large number of synthetic images need to be generated, which is the case when many different attribute combinations need to be tested. We propose an image classifier benchmarking method as an iterative process that alternates image generation, classifier evaluation, and attribute selection. This method efficiently explores the attributes that ultimately lead to poor behavior detection.
Code (0)
등록된 구현이 없습니다.
Tasks
AttributeBayesian OptimizationBenchmarkingEfficient ExplorationImage GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
PARyOpt: A software for Parallel Asynchronous Remote Bayesian Optimization
PARyOpt is a python based implementation of the Bayesian optimization routine designed for remote and asynchronous function evaluations. Bayesian optimization is especially attractive for computational optimization due t…
Bayesian OptimizationDistributed ComputingSurface ReconstructionTesting Neural Networks via Bayesian-Guided Exploration of Decision Landscapes
As neural networks are increasingly deployed in safety-critical domains, testing is essential to evaluate and improve their reliability. Existing testing methods, whether black-box or white-box, primarily use global muta…
CBOL-Tuner: Classifier-pruned Bayesian optimization to explore temporally structured latent spaces for particle accelerator tuning
Complex dynamical systems, such as particle accelerators, often require complicated and time-consuming tuning procedures for optimal performance. It may also be required that these procedures estimate the optimal system …
Bayesian OptimizationEfficient Explorationglobal-optimizationparameter estimationPredicting Cascading Failures in Power Systems using Machine Learning
Cascading failure studies help assess and enhance the robustness of power systems against severe power outages. Onset time is a critical parameter in the analysis and management of power system stability and reliability,…
Bayesian OptimizationManagementMulti-class ClassificationExcursion Search for Constrained Bayesian Optimization under a Limited Budget of Failures
When learning to ride a bike, a child falls down a number of times before achieving the first success. As falling down usually has only mild consequences, it can be seen as a tolerable failure in exchange for a faster le…
Bayesian Optimization