Interpretation of the Transformer and Improvement of the Extractor

2023-11-21 · Zhe Chen

It has been over six years since the Transformer architecture was put forward. Surprisingly, the vanilla Transformer architecture is still widely used today. One reason is that the lack of deep understanding and comprehensive interpretation of the Transformer architecture makes it more challenging to improve the Transformer architecture. In this paper, we first interpret the Transformer architecture comprehensively in plain words based on our understanding and experiences. The interpretations are further proved and verified. These interpretations also cover the Extractor, a family of drop-in replacements for the multi-head self-attention in the Transformer architecture. Then, we propose an improvement on a type of the Extractor that outperforms the self-attention, without introducing additional trainable parameters. Experimental results demonstrate that the improved Extractor performs even better, showing a way to improve the Transformer architecture.

📄 PDF Abstract BibTeX arXiv:2311.12678

Code (1)

fabianwinter93/JAX/tree/main/SuperHighPerformanceExtractor jax

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Multi-Head Attention 설명 없음

Residual Connection 설명 없음

VB calibration to improve the interface between phone recognizer and i-vector extractor

2015-10-12 · Niko Brümmer

The EM training algorithm of the classical i-vector extractor is often incorrectly described as a maximum-likelihood method. The i-vector model is however intractable: the likelihood itself and the hidden-variable poster…

Transformer Assisted Convolutional Network for Cell Instance Segmentation

2021-10-05 · Deepanshu Pandey, Pradyumna Gupta, Sumit Bhattacharya, Aman Sinha 외

Region proposal based methods like R-CNN and Faster R-CNN models have proven to be extremely successful in object detection and segmentation tasks. Recently, Transformers have also gained popularity in the domain of Comp…

Instance Segmentation object-detection Object Detection Region Proposal+1

Vision Transformer-based Adversarial Domain Adaptation

2024-04-24 · Yahan Li, Yuan Wu

Unsupervised domain adaptation (UDA) aims to transfer knowledge from a labeled source domain to an unlabeled target domain. The most recent UDA methods always resort to adversarial training to yield state-of-the-art resu…

Domain Adaptation image-classification Image Classification object-detection+3

Attention Is Not All You Need Anymore

2023-08-15 · Zhe Chen

In recent years, the popular Transformer architecture has achieved great success in many application areas, including natural language processing and computer vision. Many existing works aim to reduce the computational a…

All Text Generation

GCI: A (G)raph (C)oncept (I)nterpretation Framework

2023-02-09 · Dmitry Kazhdan, Botty Dimanov, Lucie Charlotte Magister, Pietro Barbiero 외

Explainable AI (XAI) underwent a recent surge in research on concept extraction, focusing on extracting human-interpretable concepts from Deep Neural Networks. An important challenge facing concept extraction approaches …

Explainable Artificial Intelligence (XAI)Molecular Property Prediction Prediction Property Prediction