Language Modelling on enwik8

42개 결과 · ⬇ CSV · JSON

Bit per Character (BPC)

Rank	Model	Bit per Character (BPC)	Number of params	Extra Training Data	Paper	Code	Year
1	GPT-2 (48 layers, h=1600)	0.93	1542M	✓	Language Models are Unsupervised Multitask Learners	huggingface/transformers · openai/gpt-2 · PaddlePaddle/PaddleNLP · +18	2019
2	Transformer-XL (24 layers, RMS dynamic eval, decay)	0.940	277M	✓	Dynamic Evaluation of Transformer Language Models	benkrause/dynamiceval-transformer	2019
2	Focus	0.940	22M		Focus Your Attention (with Adaptive IIR Filters)		2023
4	Expire-Span (24 layers)	0.95	208M		Not All Memories are Created Equal: Learning to Forget by Expiring	facebookresearch/transformer-sequential	2021
4	SRU++ Large	0.95	195M		When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute	asappresearch/sru	2021
6	Feedback Transformer	0.96	77M		Addressing Some Limitations of Transformers with Feedback Memory	labmlai/annotated_deep_learning_paper_implementations · facebookresearch/transformer-sequential · lucidrains/feedback-transformer-pytorch · +1	2020
7	Sandwich Transformer (adaptive span)	0.968	209M		Improving Transformer Models by Reordering their Sublayers	ofirpress/sandwich_transformer · JunnYu/x-transformers-paddle	2019
8	Compressive Transformer (24 layers)	0.97	277M		Compressive Transformers for Long-Range Sequence Modelling	labmlai/annotated_deep_learning_paper_implementations · google-deepmind/pg19 · deepmind/pg19 · +3	2019
8	Transformer-LS (large)	0.97	110M		Long-Short Transformer: Efficient Transformers for Language and Vision	keonlee9420/Comprehensive-Transformer-TTS · NVIDIA/transformer-ls · lucidrains/long-short-transformer	2021
8	SRU++ Base	0.97	108M		When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute	asappresearch/sru	2021
11	Transformer (24 layers, 8k adaptive span)	0.98	209M		Adaptive Attention Span in Transformers	facebookresearch/adaptive-span · jerrodparker20/adaptive-transformers-in-rl · prajjwal1/fluence · +5	2019
12	Transformer-XL (24 layers)	0.99	277M		Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	huggingface/transformers · labmlai/annotated_deep_learning_paper_implementations · NVIDIA/DeepLearningExamples · +34	2019
12	Longformer (30 layers, h=512)	0.99	102M		Longformer: The Long-Document Transformer	huggingface/transformers · mistralai/mistral-src · facebookresearch/xformers · +19	2020
12	Sparse Transformer (30 layers, fixed attn)	0.99	95M		Generating Long Sequences with Sparse Transformers	mistralai/mistral-src · openai/sparse_attention · wilson1yan/VideoGPT · +4	2019
12	Routing Transformer (12 layers)	0.99	–		Efficient Content-Based Sparse Attention with Routing Transformers	lucidrains/local-attention · lucidrains/routing-transformer	2020
12	Transformer-LS (small)	0.99	–		Long-Short Transformer: Efficient Transformers for Language and Vision	keonlee9420/Comprehensive-Transformer-TTS · NVIDIA/transformer-ls · lucidrains/long-short-transformer	2021
17	Hourglass	0.997	–		Hierarchical Transformers Are More Efficient Language Models	labmlai/annotated_deep_learning_paper_implementations · google/trax · lucidrains/hourglass-transformer-pytorch	2021
18	Longformer (12 layers, h=512)	1.00	41M		Longformer: The Long-Document Transformer	huggingface/transformers · mistralai/mistral-src · facebookresearch/xformers · +19	2020
19	All-attention network (18 layers)	1.01	39M		Augmenting Self-attention with Persistent Memory	lucidrains/x-transformers · facebookresearch/adaptive-span	2019
20	Transformer (12 layers, 8k adaptive span)	1.02	39M		Adaptive Attention Span in Transformers	facebookresearch/adaptive-span · jerrodparker20/adaptive-transformers-in-rl · prajjwal1/fluence · +5	2019