홈 › Papers

A Simple and Effective Positional Encoding for Transformers

2021-04-18 · EMNLP 2021 11 · Pu-Chin Chen, Henry Tsai, Srinadh Bhojanapalli, Hyung Won Chung, Yin-Wen Chang, Chun-Sung Ferng

Transformer models are permutation equivariant. To supply the order and type information of the input tokens, position and segment embeddings are usually added to the input. Recent works proposed variations of positional encodings with relative position encodings achieving better performance. Our analysis shows that the gain actually comes from moving positional information to attention layer from the input. Motivated by this, we introduce Decoupled Positional Attention for Transformers (DIET), a simple yet effective mechanism to encode position and segment information into the Transformer models. The proposed method has faster training and inference time, while achieving competitive performance on GLUE, XTREME and WMT benchmarks. We further generalize our method to long-range transformers and show performance gain.

📄 PDF Abstract BibTeX arXiv:2104.08698

Code (0)

등록된 구현이 없습니다.

Tasks

Position

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Relative Position Encodings Relative Position Encodings are a type of position embeddings for Transformer-based models that attempts to exploit…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Residual Connection 설명 없음

Learnable Spatial-Temporal Positional Encoding for Link Prediction

2025-06-10 · Katherine Tieu, Dongqi Fu, Zihao Li, Ross Maciejewski 외

Accurate predictions rely on the expressiveness power of graph deep learning frameworks like graph neural networks and graph transformers, where a positional encoding mechanism has become much more indispensable in recen…

Link Prediction Prediction

Randomized Positional Encodings Boost Length Generalization of Transformers

2023-05-26 · Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya 외

Transformers have impressive generalization capabilities on tasks with a fixed context length. However, they fail to generalize to sequences of arbitrary length, even for seemingly simple tasks such as duplicating a stri…

Conditional Positional Encodings for Vision Transformers

2021-02-22 · Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang 외

We propose a conditional positional encoding (CPE) scheme for vision Transformers. Unlike previous fixed or learnable positional encodings, which are pre-defined and independent of input tokens, CPE is dynamically genera…

AutoML Classification General Classification image-classification+3

Linearized Relative Positional Encoding

2023-07-18 · Zhen Qin, Weixuan Sun, Kaiyue Lu, Hui Deng 외

Relative positional encoding is widely used in vanilla and linear transformers to represent positional information. However, existing encoding methods of a vanilla transformer are not always directly applicable to a line…

image-classification Image Classification Language Modeling Language Modelling+2

Comparing Graph Transformers via Positional Encodings

2024-02-22 · Mitchell Black, Zhengchao Wan, Gal Mishne, Amir Nayyeri 외

The distinguishing power of graph transformers is closely tied to the choice of positional encoding: features used to augment the base transformer with information about the graph. There are two primary types of position…

Navigate