paper-with-me

Language Modelling 벤치마크

Language Modelling on enwik8

42개 결과 · ⬇ CSV · JSON

Bit per Character (BPC)

0.93 1.115 1.3 1.485 1.67 2013-08 2026-09 LSTM (7 layers) — 1.67 (2013-08-04) Recurrent Highway Networks — 1.27 (2016-07-12) LN HM-LSTM — 1.32 (2016-09-06) Large mLSTM — 1.24 (2016-09-26) Hypernetworks — 1.34 (2016-09-27) ByteNet — 1.31 (2016-10-31) Large FS-LSTM-4 — 1.25 (2017-05-24) AWD-LSTM (3 layers) — 1.232 (2018-03-22) Transformer (64 layers) — 1.06 (2018-08-09) 64-layer Character Transformer Model — 1.11 (2018-08-09) Transformer-XL (24 layers) — 0.99 (2019-01-09) Transformer-XL (18 layers) — 1.03 (2019-01-09) Transformer-XL (12 layers) — 1.06 (2019-01-09) GPT-2 (48 layers, h=1600) — 0.93 (2019-02-14) Transformer-XL (24 layers, RMS dynamic eval, decay) — 0.94 (2019-04-17) Sparse Transformer (30 layers, fixed attn) — 0.99 (2019-04-23) Transformer (24 layers, 8k adaptive span) — 0.98 (2019-05-19) Transformer (12 layers, 8k adaptive span) — 1.02 (2019-05-19) All-attention network (18 layers) — 1.01 (2019-07-02) Mogrifier LSTM — 1.146 (2019-09-04) LSTM — 1.195 (2019-09-04) Sandwich Transformer (adaptive span) — 0.968 (2019-11-10) BP-Transformer (12 layers) — 1.02 (2019-11-11) Compressive Transformer (24 layers) — 0.97 (2019-11-13) SHA-RNN (4 layers, h=1024, attention head per layer) — 1.068 (2019-11-26) SHA-RNN (4 layers, h=1024, single attention head) — 1.076 (2019-11-26) SHA-LSTM (4 layers, h=1024, no attention head) — 1.33 (2019-11-26) Feedback Transformer — 0.96 (2020-02-21) Routing Transformer (12 layers) — 0.99 (2020-03-12) Longformer (30 layers, h=512) — 0.99 (2020-04-10) Longformer (12 layers, h=512) — 1.0 (2020-04-10) Cluster-Former (#C=512) — 1.22 (2020-09-13) SRU++ Large — 0.95 (2021-02-24) SRU++ Base — 0.97 (2021-02-24) Expire-Span (24 layers) — 0.95 (2021-05-13) Transformer-LS (large) — 0.97 (2021-07-05) Transformer-LS (small) — 0.99 (2021-07-05) Hourglass — 0.997 (2021-10-26) Focus — 0.94 (2023-05-24) Transformer+SSA — 1.024 (2023-06-02) Skip Cross-Head Transformer-XL — 1.033 (2023-11-14) LSTM (7 layers) — 1.67 (2013-08-04)
RankModel Bit per Character (BPC)Number of params Extra Training Data PaperCodeYear
1 GPT-2 (48 layers, h=1600) 0.931542M ✓ Language Models are Unsupervised Multitask Learners huggingface/transformers · openai/gpt-2 · PaddlePaddle/PaddleNLP · +18 2019
2 Transformer-XL (24 layers, RMS dynamic eval, decay) 0.940277M ✓ Dynamic Evaluation of Transformer Language Models benkrause/dynamiceval-transformer 2019
2 Focus 0.94022M Focus Your Attention (with Adaptive IIR Filters) 2023
4 Expire-Span (24 layers) 0.95208M Not All Memories are Created Equal: Learning to Forget by Expiring facebookresearch/transformer-sequential 2021
4 SRU++ Large 0.95195M When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute asappresearch/sru 2021
6 Feedback Transformer 0.9677M Addressing Some Limitations of Transformers with Feedback Memory labmlai/annotated_deep_learning_paper_implementations · facebookresearch/transformer-sequential · lucidrains/feedback-transformer-pytorch · +1 2020
7 Sandwich Transformer (adaptive span) 0.968209M Improving Transformer Models by Reordering their Sublayers ofirpress/sandwich_transformer · JunnYu/x-transformers-paddle 2019
8 Compressive Transformer (24 layers) 0.97277M Compressive Transformers for Long-Range Sequence Modelling labmlai/annotated_deep_learning_paper_implementations · google-deepmind/pg19 · deepmind/pg19 · +3 2019
8 Transformer-LS (large) 0.97110M Long-Short Transformer: Efficient Transformers for Language and Vision keonlee9420/Comprehensive-Transformer-TTS · NVIDIA/transformer-ls · lucidrains/long-short-transformer 2021
8 SRU++ Base 0.97108M When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute asappresearch/sru 2021
11 Transformer (24 layers, 8k adaptive span) 0.98209M Adaptive Attention Span in Transformers facebookresearch/adaptive-span · jerrodparker20/adaptive-transformers-in-rl · prajjwal1/fluence · +5 2019
12 Transformer-XL (24 layers) 0.99277M Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context huggingface/transformers · labmlai/annotated_deep_learning_paper_implementations · NVIDIA/DeepLearningExamples · +34 2019
12 Longformer (30 layers, h=512) 0.99102M Longformer: The Long-Document Transformer huggingface/transformers · mistralai/mistral-src · facebookresearch/xformers · +19 2020
12 Sparse Transformer (30 layers, fixed attn) 0.9995M Generating Long Sequences with Sparse Transformers mistralai/mistral-src · openai/sparse_attention · wilson1yan/VideoGPT · +4 2019
12 Routing Transformer (12 layers) 0.99– Efficient Content-Based Sparse Attention with Routing Transformers lucidrains/local-attention · lucidrains/routing-transformer 2020
12 Transformer-LS (small) 0.99– Long-Short Transformer: Efficient Transformers for Language and Vision keonlee9420/Comprehensive-Transformer-TTS · NVIDIA/transformer-ls · lucidrains/long-short-transformer 2021
17 Hourglass 0.997– Hierarchical Transformers Are More Efficient Language Models labmlai/annotated_deep_learning_paper_implementations · google/trax · lucidrains/hourglass-transformer-pytorch 2021
18 Longformer (12 layers, h=512) 1.0041M Longformer: The Long-Document Transformer huggingface/transformers · mistralai/mistral-src · facebookresearch/xformers · +19 2020
19 All-attention network (18 layers) 1.0139M Augmenting Self-attention with Persistent Memory lucidrains/x-transformers · facebookresearch/adaptive-span 2019
20 Transformer (12 layers, 8k adaptive span) 1.0239M Adaptive Attention Span in Transformers facebookresearch/adaptive-span · jerrodparker20/adaptive-transformers-in-rl · prajjwal1/fluence · +5 2019
20 BP-Transformer (12 layers) 1.0238M BP-Transformer: Modelling Long-Range Context via Binary Partitioning dmlc/dgl · yzh119/BPT 2019
22 Transformer+SSA 1.024– The Information Pathways Hypothesis: Transformers are Dynamic Self-Ensembles shamim-hussain/ssa 2023
23 Transformer-XL (18 layers) 1.0388M Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context huggingface/transformers · labmlai/annotated_deep_learning_paper_implementations · NVIDIA/DeepLearningExamples · +34 2019
24 Skip Cross-Head Transformer-XL 1.03341M Memory-efficient Stochastic methods for Memory-based Transformers vishwajit-vishnu/memory-efficient-stochastic-methods-for-memory-based-transformers 2023
25 Transformer (64 layers) 1.06235M Character-Level Language Modeling with Deeper Self-Attention facebookresearch/code-prediction-transformer 2018
25 Transformer-XL (12 layers) 1.0641M Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context huggingface/transformers · labmlai/annotated_deep_learning_paper_implementations · NVIDIA/DeepLearningExamples · +34 2019
27 SHA-RNN (4 layers, h=1024, attention head per layer) 1.06854M Single Headed Attention RNN: Stop Thinking With Your Head Smerity/sha-rnn · floleuerer/fastai_ulmfit · saattrupdan/scholarly · +2 2019
28 SHA-RNN (4 layers, h=1024, single attention head) 1.07652M Single Headed Attention RNN: Stop Thinking With Your Head Smerity/sha-rnn · floleuerer/fastai_ulmfit · saattrupdan/scholarly · +2 2019
29 64-layer Character Transformer Model 1.1144M Character-Level Language Modeling with Deeper Self-Attention facebookresearch/code-prediction-transformer 2018
30 Mogrifier LSTM 1.14648M Mogrifier LSTM deepmind/lamb · RMichaelSwan/MogrifierLSTM · microcoder-py/mogrifier-lstm 2019
31 LSTM 1.19548M Mogrifier LSTM deepmind/lamb · RMichaelSwan/MogrifierLSTM · microcoder-py/mogrifier-lstm 2019
32 Cluster-Former (#C=512) 1.22– Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding 2020
33 AWD-LSTM (3 layers) 1.23247M An Analysis of Neural Language Modeling at Multiple Scales salesforce/awd-lstm-lm · Han-JD/GRU-D · jb33k/awd-lstm-lm-ThinkNet · +9 2018
34 Large mLSTM 1.2446M Multiplicative LSTM for sequence modelling astakara48/python_project 2016
35 Large FS-LSTM-4 1.2547M Fast-Slow Recurrent Neural Networks amujika/Fast-Slow-LSTM 2017
36 Recurrent Highway Networks 1.2746M Recurrent Highway Networks labmlai/annotated_deep_learning_paper_implementations · julian121266/RecurrentHighwayNetworks · jzilly/RecurrentHighwayNetworks · +3 2016
37 ByteNet 1.31– Neural Machine Translation in Linear Time paarthneekhara/byteNet-tensorflow · microsoft/protein-sequence-models · randomrandom/deep-atrous-cnn-sentiment · +8 2016
38 LN HM-LSTM 1.3235M Hierarchical Multiscale Recurrent Neural Networks bolducp/hierarchical-rnn · kaiu85/hm-rnn · nikolasthuesen/HMLSTM 2016
39 SHA-LSTM (4 layers, h=1024, no attention head) 1.3351M Single Headed Attention RNN: Stop Thinking With Your Head Smerity/sha-rnn · floleuerer/fastai_ulmfit · saattrupdan/scholarly · +2 2019
40 Hypernetworks 1.3427M HyperNetworks labmlai/annotated_deep_learning_paper_implementations · g1910/HyperNetworks · tjuhaoxiaotian/pymarl3 · +7 2016
41 LSTM (7 layers) 1.67– Generating Sequences With Recurrent Neural Networks karpathy/char-rnn · sjvasquez/handwriting-synthesis · karpathy/makemore · +56 2013
42 All-attention network (36 layers) –114M Augmenting Self-attention with Persistent Memory lucidrains/x-transformers · facebookresearch/adaptive-span 2019
1–42 / 42 페이지당 10 20 50 100