Towards Good Practices in Evaluating Transfer Adversarial Attacks
Transfer adversarial attacks raise critical security concerns in real-world, black-box scenarios. However, the actual progress of this field is difficult to assess due to two common limitations in existing evaluations. First, different methods are often not systematically and fairly evaluated in a one-to-one comparison. Second, only transferability is evaluated but another key attack property, stealthiness, is largely overlooked. In this work, we design good practices to address these limitations, and we present the first comprehensive evaluation of transfer attacks, covering 23 representative attacks against 9 defenses on ImageNet. In particular, we propose to categorize existing attacks into five categories, which enables our systematic category-wise analyses. These analyses lead to new findings that even challenge existing knowledge and also help determine the optimal attack hyperparameters for our attack-wise comprehensive evaluation. We also pay particular attention to stealthiness, by adopting diverse imperceptibility metrics and looking into new, finer-grained characteristics. Overall, our new insights into transferability and stealthiness lead to actionable good practices for future evaluations.
Code (1)
Similar Papers 제목 키워드 기반
Devling into Adversarial Transferability on Image Classification: Review, Benchmark, and Evaluation
Adversarial transferability refers to the capacity of adversarial examples generated on the surrogate model to deceive alternate, unexposed victim models. This property eliminates the need for direct access to the victim…
Image ClassificationOn Evaluating Adversarial Robustness
Correctly evaluating defenses against adversarial examples has proven to be extremely difficult. Despite the significant amount of recent work attempting to design defenses that withstand adaptive attacks, few have succe…
Adversarial AttackAdversarial DefenseAdversarial RobustnessOn Evaluating Adversarial Robustness of Chest X-ray Classification: Pitfalls and Best Practices
Vulnerability to adversarial attacks is a well-known weakness of Deep Neural Networks. While most of the studies focus on natural images with standardized benchmarks like ImageNet and CIFAR, little research has considere…
Adversarial RobustnessClassificationMedical DiagnosisX-ray ClassificationI Stolenly Swear That I Am Up to (No) Good: Design and Evaluation of Model Stealing Attacks
Model stealing attacks endanger the confidentiality of machine learning models offered as a service. Although these models are kept secret, a malicious party can query a model to label data samples and train their own su…
Image ClassificationRecent improvements of ASR models in the face of adversarial attacks
Like many other tasks involving neural networks, Speech Recognition models are vulnerable to adversarial attacks. However recent research has pointed out differences between attacks and defenses on ASR models compared to…
speech-recognitionSpeech Recognition