Adversarial Token Attacks on Vision Transformers

2021-10-08 · Ameya Joshi, Gauri Jagatap, Chinmay Hegde

Vision transformers rely on a patch token based self attention mechanism, in contrast to convolutional networks. We investigate fundamental differences between these two families of models, by designing a block sparsity based adversarial token attack. We probe and analyze transformer as well as convolutional models with token attacks of varying patch sizes. We infer that transformer models are more sensitive to token attacks than convolutional models, with ResNets outperforming Transformer models by up to $\sim30\%$ in robust accuracy for single token attacks.

📄 PDF Abstract BibTeX arXiv:2110.04337

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Residual Connection 설명 없음

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Adam 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Transferable Adversarial Attacks on Vision Transformers with Token Gradient Regularization

2023-03-28 · CVPR 2023 1 · Jianping Zhang, Yizhan Huang, Weibin Wu, Michael R. Lyu

Vision transformers (ViTs) have been successfully deployed in a variety of computer vision tasks, but they are still vulnerable to adversarial samples. Transfer-based attacks use a local model to generate adversarial sam…

Attention Deficit is Ordered! Fooling Deformable Vision Transformers with Collaborative Adversarial Patches

2023-11-21 · Quazi Mishkatul Alam, Bilel Tarchoun, Ihsen Alouani, Nael Abu-Ghazaleh

The latest generation of transformer-based vision models has proven to be superior to Convolutional Neural Network (CNN)-based models across several vision tasks, largely attributed to their remarkable prowess in relatio…

object-detection Object Detection

Robustness Tokens: Towards Adversarial Robustness of Transformers

2025-03-13 · Brian Pulfer, Yury Belousov, Slava Voloshynovskiy

Recently, large pre-trained foundation models have become widely adopted by machine learning practitioners for a multitude of tasks. Given that such models are publicly available, relying on their use as backbone models …

Adversarial Robustness

On Improving Adversarial Transferability of Vision Transformers

2021-06-08 · ICLR 2022 4 · Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Fahad Shahbaz Khan 외

Vision transformers (ViTs) process input images as sequences of patches via self-attention; a radically different architecture than convolutional neural networks (CNNs). This makes it interesting to study the adversarial…

Adversarial Attack

Random Entangled Tokens for Adversarially Robust Vision Transformer

2024-01-01 · CVPR 2024 1 · Huihui Gong, Minjing Dong, Siqi Ma, Seyit Camtepe 외

Vision Transformers (ViTs) have emerged as a compelling alternative to Convolutional Neural Networks (CNNs) in the realm of computer vision showcasing tremendous potential. However recent research has unveiled a susc…