paper-with-me

홈 › Papers

A Study of the Attention Abnormality in Trojaned BERTs

2022-05-13 · NAACL 2022 7 · Weimin Lyu, Songzhu Zheng, Tengfei Ma, Chao Chen

Trojan attacks raise serious security concerns. In this paper, we investigate the underlying mechanism of Trojaned BERT models. We observe the attention focus drifting behavior of Trojaned models, i.e., when encountering an poisoned input, the trigger token hijacks the attention focus regardless of the context. We provide a thorough qualitative and quantitative analysis of this phenomenon, revealing insights into the Trojan mechanism. Based on the observation, we propose an attention-based Trojan detector to distinguish Trojaned models from clean ones. To the best of our knowledge, this is the first paper to analyze the Trojan mechanism and to develop a Trojan detector based on the transformer's attention.

📄 PDF Abstract BibTeX arXiv:2205.08305

Code (1)

weimin17/attention_abnormality_in_trojaned_berts 공식 구현 pytorch

Methods 이 논문이 사용한 방법론

Attention 설명 없음
Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.
Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…
Multi-Head Attention 설명 없음
Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…
Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…
Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…
Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Similar Papers 제목 키워드 기반

A Study of the Attention Abnormality in Trojaned BERTs

2022-01-16 · ACL ARR January 2022 1 · Anonymous

Trojan attacks raise serious security concerns. In this paper, we investigate the underlying mechanism of Trojaned BERT models. We observe the attention focus drifting behavior of Trojaned models, i.e., when encountering…

Topological Detection of Trojaned Neural Networks

2021-06-11 · NeurIPS 2021 12 · Songzhu Zheng, Yikai Zhang, Hubert Wagner, Mayank Goswami 외

Deep neural networks are known to have security issues. One particular threat is the Trojan attack. It occurs when the attackers stealthily manipulate the model's behavior through Trojaned training samples, which can lat…

On Trojan Signatures in Large Language Models of Code

2024-02-23 · Aftab Hussain, Md Rafiqul Islam Rabin, Mohammad Amin Alipour

Trojan signatures, as described by Fields et al. (2021), are noticeable differences in the distribution of the trojaned class parameters (weights) and the non-trojaned class parameters of the trojaned model, that can be …

Binary ClassificationDefect Detection

Toward a Taxonomy and Computational Models of Abnormalities in Images

2015-12-04 · Babak Saleh, Ahmed Elgammal, Jacob Feldman, Ali Farhadi

The human visual system can spot an abnormal image, and reason about what makes it strange. This task has not received enough attention in computer vision. In this paper we study various types of atypicalities in images …

Scanning Trojaned Models Using Out-of-Distribution Samples

2025-01-28 · Hossein Mirzaei, Ali Ansari, Bahar Dibaei Nia, Mojtaba Nafez 외

Scanning for trojan (backdoor) in deep neural networks is crucial due to their significant real-world applications. There has been an increasing focus on developing effective general trojan scanning methods across variou…

Backdoor Attack