Introduction to Transformers: an NLP Perspective

2023-11-29 · Tong Xiao, Jingbo Zhu

Transformers have dominated empirical machine learning models of natural language processing. In this paper, we introduce basic concepts of Transformers and present key techniques that form the recent advances of these models. This includes a description of the standard Transformer architecture, a series of model refinements, and common applications. Given that Transformers and related deep learning techniques might be evolving in ways we have never seen, we cannot dive into all the model details or cover all the technical areas. Instead, we focus on just those concepts that are helpful for gaining a good understanding of Transformers and their variants. We also summarize the key ideas that impact this field, thereby yielding some insights into the strengths and limitations of these models.

📄 PDF Abstract BibTeX arXiv:2311.17633

Code (1)

NiuTrans/Introduction-to-Transformers 공식 구현 tf

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Verification of Neural Networks (Lecture Notes)

2026-04-28 · Benedikt Bollig arxiv

These lecture notes provide an introduction to the verification of neural networks from a theoretical perspective. We discuss feed-forward neural networks, recurrent neural networks, attention mechanisms, and transformer…

Were RNNs All We Needed?

2024-10-02 · Leo Feng, Frederick Tung, Mohamed Osama Ahmed, Yoshua Bengio 외

The introduction of Transformers in 2017 reshaped the landscape of deep learning. Originally proposed for sequence modelling, Transformers have since achieved widespread success across various domains. However, the scala…

All Mamba

Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives

2022-06-02 · Jun Li, Junyu Chen, Yucheng Tang, Ce Wang 외

Transformer, the latest technological advance of deep learning, has gained prevalence in natural language processing or computer vision. Since medical imaging bear some resemblance to computer vision, it is natural to in…

Image Segmentation Medical Image Segmentation Semantic Segmentation

An Introduction to Transformers

2023-04-20 · Richard E. Turner

The transformer is a neural network component that can be used to learn useful representations of sequences or sets of data-points. The transformer has driven recent advances in natural language processing, computer visi…

Sequence Complementor: Complementing Transformers For Time Series Forecasting with Learnable Sequences

2025-01-06 · Xiwen Chen, Peijie Qiu, Wenhui Zhu, Huayu Li 외

Since its introduction, the transformer has shifted the development trajectory away from traditional models (e.g., RNN, MLP) in time series forecasting, which is attributed to its ability to capture global dependencies w…

Time Series Time Series Forecasting