Arbitrary-Length Generalization for Addition in a Tiny Transformer

2024-05-31 · Alexandre Galvao Patriota

This paper introduces a novel training methodology that enables a Transformer model to generalize the addition of two-digit numbers to numbers with unseen lengths of digits. The proposed approach employs an autoregressive generation technique, processing from right to left, which mimics a common manual method for adding large numbers. To the best of my knowledge, this methodology has not been previously explored in the literature. All results are reproducible, and the corresponding R code is available at github.com/AGPatriota/ALGA-R/.

📄 PDF Abstract BibTeX arXiv:2406.00075

Code (1)

agpatriota/alga-r 공식 구현 pytorch

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Adam 설명 없음

Attention 설명 없음

Residual Connection 설명 없음

Universal Length Generalization with Turing Programs

2024-07-03 · Kaiying Hou, David Brandfonbrener, Sham Kakade, Samy Jelassi 외

Length generalization refers to the ability to extrapolate from short training sequences to long test sequences and is a challenge for current large language models. While prior work has proposed some architecture or dat…

Dynamic-TinyBERT: Boost TinyBERT's Inference Efficiency by Dynamic Sequence Length

2021-11-18 · Shira Guskin, Moshe Wasserblat, Ke Ding, Gyuwan Kim

Limited computational budgets often prevent transformers from being used in production and from having their high accuracy utilized. TinyBERT addresses the computational efficiency by self-distilling BERT into a smaller …

Computational Efficiency Hyperparameter Optimization Question Answering

Arithmetic Transformers Can Length-Generalize in Both Operand Length and Count

2024-10-21 · Hanseul Cho, Jaeyoung Cha, Srinadh Bhojanapalli, Chulhee Yun

Transformers often struggle with length generalization, meaning they fail to generalize to sequences longer than those encountered during training. While arithmetic tasks are commonly used to study length generalization,…

Position

QuaLA-MiniLM: a Quantized Length Adaptive MiniLM

2022-10-31 · Shira Guskin, Moshe Wasserblat, Chang Wang, Haihao Shen

Limited computational budgets often prevent transformers from being used in production and from having their high accuracy utilized. A knowledge distillation approach addresses the computational efficiency by self-distil…

Computational Efficiency Knowledge Distillation Quantization Question Answering

Softmax Transformers are Turing-Complete

2025-11-25 · Hongjian Jiang, Michael Hahn, Georg Zetzsche, Anthony Widjaja Lin arxiv

Hard attention Chain-of-Thought (CoT) transformers are known to be Turing-complete. However, it is an open problem whether softmax attention Chain-of-Thought (CoT) transformers are Turing-complete. In this paper, we prov…

Arithmetic Reasoning