Language Modelling on Text8

24개 결과 · ⬇ CSV · JSON

Bit per Character (BPC)

Rank	Model	Bit per Character (BPC)	Number of params	Extra Training Data	Paper	Code	Year
1	GPT-2	0.98	1542M	✓	Language Models are Unsupervised Multitask Learners	huggingface/transformers · openai/gpt-2 · PaddlePaddle/PaddleNLP · +18	2019
1	Focus	0.98	22M		Focus Your Attention (with Adaptive IIR Filters)		2023
3	Transformer-XL + RMS dynamic eval + decay	1.038	277M		Dynamic Evaluation of Transformer Language Models	benkrause/dynamiceval-transformer	2019
4	24L Transformer + 8K adaptive span	1.07	209M		Adaptive Attention Span in Transformers	facebookresearch/adaptive-span · jerrodparker20/adaptive-transformers-in-rl · prajjwal1/fluence · +5	2019
5	Transformer-XL - 24 layers	1.08	277M		Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	huggingface/transformers · labmlai/annotated_deep_learning_paper_implementations · NVIDIA/DeepLearningExamples · +34	2019
5	All-attention network - 36 layers	1.08	114M		Augmenting Self-attention with Persistent Memory	lucidrains/x-transformers · facebookresearch/adaptive-span	2019
7	Transformer-LS (small)	1.09	–		Long-Short Transformer: Efficient Transformers for Language and Vision	keonlee9420/Comprehensive-Transformer-TTS · NVIDIA/transformer-ls · lucidrains/long-short-transformer	2021
8	12L Transformer + 8K adaptive span	1.11	38M		Adaptive Attention Span in Transformers	facebookresearch/adaptive-span · jerrodparker20/adaptive-transformers-in-rl · prajjwal1/fluence · +5	2019
8	All-attention network - 18 layers	1.11	38M		Augmenting Self-attention with Persistent Memory	lucidrains/x-transformers · facebookresearch/adaptive-span	2019
8	BP-Transformer - 12 Layers	1.11	–		BP-Transformer: Modelling Long-Range Context via Binary Partitioning	dmlc/dgl · yzh119/BPT	2019
11	64-layer Character Transformer Model	1.13	235M		Character-Level Language Modeling with Deeper Self-Attention	facebookresearch/code-prediction-transformer	2018
12	GAM-RHN-10	1.157	44.7M		Recurrent Highway Networks with Grouped Auxiliary Memory	WilliamRo/gam_rhn · MindCode-4/code-8 · MindSpore-scientific/code-5 · +1	2019
13	12-layer Character Transformer Model	1.18	44M		Character-Level Language Modeling with Deeper Self-Attention	facebookresearch/code-prediction-transformer	2018
13	PAR Transformer 24B	1.18	–		Pay Attention when Required	NVIDIA/DeepLearningExamples · Jmkernes/PAR-Transformer-XL	2020
15	mLSTM + dynamic eval	1.19	45M		Dynamic Evaluation of Neural Sequence Models	benkrause/dynamic-evaluation · benkrause/dynamiceval-transformer · sacmehta/PRU	2017
16	Bipartite flows (8 flows)	1.23	–		Discrete Flows: Invertible Generative Models of Discrete Data	google/edward2 · TrentBrick/PyTorchDiscreteFlows	2019
17	Large RHN	1.27	46M		Recurrent Highway Networks	labmlai/annotated_deep_learning_paper_implementations · julian121266/RecurrentHighwayNetworks · jzilly/RecurrentHighwayNetworks · +3	2016
17	Large mLSTM +emb +WN +VD	1.27	45M		Multiplicative LSTM for sequence modelling	astakara48/python_project	2016
19	LayerNorm HM-LSTM	1.29	35M		Hierarchical Multiscale Recurrent Neural Networks	bolducp/hierarchical-rnn · kaiu85/hm-rnn · nikolasthuesen/HMLSTM	2016
20	BN LSTM	1.36	16M		Recurrent Batch Normalization	cooijmanstim/recurrent-batch-normalization · codedecde/Recognizing-Textual-Entailment · Tetsuya-Nishikawa/ConvLSTM_DEMO	2016