Tiny Transformers Excel at Sentence Compression
It is staggering that words of the English language, which are on average represented by 5--6 bytes of ASCII, require as much as 24 kilobytes when served to large language models. We show that there is room for more information in every token embedding. We demonstrate that 1--3-layer transformers are capable of encoding and subsequently decoding standard English sentences into as little as a single 3-kilobyte token. Our work implies that even small networks can learn to construct valid English sentences and suggests the possibility of optimising large language models by moving from sub-word token embeddings towards larger fragments of text.
Code (0)
등록된 구현이 없습니다.
Tasks
SentenceSentence CompressionvalidSimilar Papers 제목 키워드 기반
Sentence Compression via DC Programming Approach
Sentence compression is an important problem in natural language processing. In this paper, we firstly establish a new sentence compression model based on the probability model and the parse tree model. Our sentence comp…
SentenceSentence CompressionA Difference-of-Convex Programming Approach With Parallel Branch-and-Bound For Sentence Compression Via A Hybrid Extractive Model
Sentence compression is an important problem in natural language processing with wide applications in text summarization, search engine and human-AI interaction system etc. In this paper, we design a hybrid extractive se…
global-optimizationLanguage ModelingLanguage ModellingSentence+2Efficient Unsupervised Sentence Compression by Fine-tuning Transformers with Reinforcement Learning
Sentence compression reduces the length of text by removing non-essential content while preserving important facts and grammaticality. Unsupervised objective driven methods for sentence compression can be used to create …
reinforcement-learningReinforcement Learning (RL)SentenceSentence Compression+1A Token-wise CNN-based Method for Sentence Compression
Sentence compression is a Natural Language Processing (NLP) task aimed at shortening original sentences and preserving their key information. Its applications can benefit many fields e.g. one can build tools for language…
SentenceSentence CompressionJoint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers
Although vision transformers (ViTs) have shown promising results in various computer vision tasks recently, their high computational cost limits their practical applications. Previous approaches that prune redundant toke…
Efficient ViTs