Improving Transformers using Faithful Positional Encoding

2024-05-15 · Tsuyoshi Idé, Jokin Labaien, Pin-Yu Chen

We propose a new positional encoding method for a neural network architecture called the Transformer. Unlike the standard sinusoidal positional encoding, our approach is based on solid mathematical grounds and has a guarantee of not losing information about the positional order of the input sequence. We show that the new encoding approach systematically improves the prediction performance in the time-series classification task.

📄 PDF Abstract BibTeX arXiv:2405.09061

Code (0)

등록된 구현이 없습니다.

Tasks

Time SeriesTime Series Classification

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Multi-Head Attention 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Position-Wise Feed-Forward Layer 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Residual Connection 설명 없음

Communicability-Inspired Positional Encoding (CIPE)

2026-06-24 · Yipeng Zhang, Zhongtian Sun, Pietro Liò, Kelin Xia arxiv

Positional encodings (PEs) are essential for Transformers. Yet designing effective PEs for non-Euclidean graphs remains challenging. Such encodings should ideally induce an Attention-Compatible Geometry for self-attentio…

Weierstrass Positional Encoding for Vision Transformers

2026-05-20 · Zhihang Xin, Rui Wang, Xitong Hu, Xiaojun Wu arxiv

Vision Transformers have achieved remarkable success in computer vision, but their common use of learnable one-dimensional positional encodings weakens the inherent two-dimensional spatial structure of images after patch…

Comparing Graph Transformers via Positional Encodings

2024-02-22 · Mitchell Black, Zhengchao Wan, Gal Mishne, Amir Nayyeri 외

The distinguishing power of graph transformers is closely tied to the choice of positional encoding: features used to augment the base transformer with information about the graph. There are two primary types of position…

Navigate

Theoretical Analysis of Hierarchical Language Recognition and Generation by Transformers without Positional Encoding

2024-10-16 · Daichi Hayakawa, Issei Sato

In this study, we provide constructive proof that Transformers can recognize and generate hierarchical language efficiently with respect to model size, even without the need for a specific positional encoding. Specifical…

Linearized Relative Positional Encoding

2023-07-18 · Zhen Qin, Weixuan Sun, Kaiyue Lu, Hui Deng 외

Relative positional encoding is widely used in vanilla and linear transformers to represent positional information. However, existing encoding methods of a vanilla transformer are not always directly applicable to a line…

image-classification Image Classification Language Modeling Language Modelling+2