How Powerful are Decoder-Only Transformer Neural Models?
In this article we prove that the general transformer neural model undergirding modern large language models (LLMs) is Turing complete under reasonable assumptions. This is the first work to directly address the Turing completeness of the underlying technology employed in GPT-x as past work has focused on the more expressive, full auto-encoder transformer architecture. From this theoretical analysis, we show that the sparsity/compressibility of the word embedding is an important consideration for Turing completeness to hold. We also show that Transformers are are a variant of B machines studied by Hao Wang.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderSimilar Papers 제목 키워드 기반
When are 1.58 bits enough? A Bottom-up Exploration of BitNet Quantization
Contemporary machine learning models, such as language models, are powerful, but come with immense resource requirements both at training and inference time. It has been shown that decoder-only language models can be tra…
DecoderQuantizationEfficient Encoder-Decoder Transformer Decoding for Decomposable Tasks
Transformer-based NLP models are powerful but have high computational costs that limit deployment. Finetuned encoder-decoder models are popular in specialized domains and can outperform larger more generalized decoder-on…
DecoderDialogue State TrackingQuestion AnsweringiSegFormer: Interactive Segmentation via Transformers with Application to 3D Knee MR Images
We propose iSegFormer, a memory-efficient transformer that combines a Swin transformer with a lightweight multilayer perceptron (MLP) decoder. With the efficient Swin transformer blocks for hierarchical self-attention an…
DecoderImage SegmentationInteractive SegmentationMedical Image Segmentation+1Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding
In this paper, we propose Shallow Aggressive Decoding (SAD) to improve the online inference efficiency of the Transformer for instantaneous Grammatical Error Correction (GEC). SAD optimizes the online inference efficienc…
DecoderGrammatical Error CorrectionEfficiently Summarizing Text and Graph Encodings of Multi-Document Clusters
This paper presents an efficient graph-enhanced approach to multi-document summarization (MDS) with an encoder-decoder Transformer model. This model is based on recent advances in pre-training both encoder and decoder on…
DecoderDocument SummarizationMulti-Document Summarization