Selective Adversarial Attacks on LLM Benchmarks
Benchmarking outcomes increasingly govern trust, selection, and deployment of LLMs, yet these evaluations remain vulnerable to semantically equivalent adversarial perturbations. Prior work on adversarial robustness in NLP has emphasized text attacks that affect many models equally, leaving open the question of whether it is possible to selectively degrade or enhance performance while minimally affecting other models. We formalize this problem and study selective adversarial attacks on MMLU - a widely used benchmark designed to measure a language model's broad general knowledge and reasoning ability across different subjects. Using canonical attacks integrated into TextAttack framework, we introduce a protocol for selectivity assessment, develop a custom constraint to increase selectivity of attacks and propose a surrogate-LLM pipeline that generates selective perturbations. Empirically, we find that selective adversarial attacks exist and can materially alter relative rankings, challenging the fairness, reproducibility, and transparency of leaderboard-driven evaluation. Our results motivate perturbation-aware reporting and robustness diagnostics for LLM evaluation and demonstrate that even subtle edits can shift comparative judgments.
Code (0)
등록된 구현이 없습니다.
Tasks
Adversarial RobustnessGeneral KnowledgeSimilar Papers 제목 키워드 기반
Resisting Adversarial Attacks using Gaussian Mixture Variational Autoencoders
Susceptibility of deep neural networks to adversarial attacks poses a major theoretical and practical challenge. All efforts to harden classifiers against such attacks have seen limited success. Two distinct categories o…
Exploring Adversarial Robustness of Vision Transformers in the Spectral Perspective
The Vision Transformer has emerged as a powerful tool for image classification tasks, surpassing the performance of convolutional neural networks (CNNs). Recently, many researchers have attempted to understand the robust…
Adversarial Robustnessimage-classificationImage ClassificationINTARG: Informed Real-Time Adversarial Attack Generation for Time-Series Regression
Time-series forecasting aims to predict future values by modeling temporal dependencies in historical observations. It is a critical component of many real-world systems, where accurate forecasts improve operational effi…
Adversarial AttackKeep on Going: Learning Robust Humanoid Motion Skills via Selective Adversarial Training
Humanoid robots are expected to operate reliably over long horizons while executing versatile whole-body skills. Yet Reinforcement Learning (RL) motion policies typically lose stability under prolonged operation, sensor/…
Reinforcement LearningAdversarial AttackRevisiting Model's Uncertainty and Confidences for Adversarial Example Detection
Security-sensitive applications that rely on Deep Neural Networks (DNNs) are vulnerable to small perturbations that are crafted to generate Adversarial Examples(AEs). The AEs are imperceptible to humans and cause DNN to …
Multi-Task LearningTransfer Learning