paper-with-me

Language Modelling 벤치마크

Language Modelling on WikiText-2

38개 결과 · ⬇ CSV · JSON

Test perplexity 낮을수록 좋음

8.21 64.85 121.5 178.1 234.8 2016-11 2026-11 Inan et al. (2016) - Variational LSTM (tied) (h=650) + augmented loss — 87.0 (2016-11-04) Inan et al. (2016) - Variational LSTM (tied) (h=650) — 87.7 (2016-11-04) Grave et al. (2016) - LSTM + continuous cache pointer — 68.9 (2016-12-13) Grave et al. (2016) - LSTM — 99.3 (2016-12-13) Melis et al. (2017) - 1-layer LSTM (tied) — 65.9 (2017-07-18) AWD-LSTM + continuous cache pointer — 52.0 (2017-08-07) AWD-LSTM — 65.8 (2017-08-07) GL-LWGC + AWD-MoS-LSTM + dynamic eval — 40.46 (2017-08-29) AWD-LSTM + dynamic eval — 44.3 (2017-09-21) AWD-LSTM 3-layer with Fraternal dropout — 64.1 (2017-10-31) AWD-LSTM-MoS + dynamic eval — 40.68 (2017-11-10) AWD-LSTM-MoS — 61.45 (2017-11-10) Past Decode Reg. + AWD-LSTM-MoS + dyn. eval. — 40.3 (2018-08-14) AWD-LSTM-DOC x5 — 53.09 (2018-08-30) AWD-LSTM-DOC — 58.03 (2018-08-30) FRAGE + AWD-LSTM-MoS + dynamic eval — 39.14 (2018-09-18) GPT-2 — 18.34 (2019-02-14) GPT-2 (large) — 19.93 (2019-02-14) GPT-2 (medium) — 22.76 (2019-02-14) GPT-2 (small) — 29.41 (2019-02-14) AWD-LSTM-DOC + Partial Shuffle — 57.85 (2019-03-11) AWD-LSTM-MoS + Partial Shuffle — 59.98 (2019-03-11) BERT-Large-CAS — 34.1 (2019-04-20) AWD-LSTM-DRILL + dynamic eval — 42.0 (2019-05-14) AWD-LSTM-DRILL — 61.9 (2019-05-14) adversarial + AWD-LSTM-MoS + dynamic eval — 38.65 (2019-06-10) Mogrifier LSTM + dynamic eval — 38.6 (2019-09-04) Mogrifier LSTM — 55.1 (2019-09-04) AWD-LSTM + ATOI — 64.73 (2019-09-18) AWD-FWM Schlag et al. (2020) — 61.65 (2020-11-16) GPT-2 (fine-tuned) — 15.17 (2021-10-16) EGRU — 68.9 (2022-06-13) SparseGPT (175B, 50% Sparsity) — 8.21 (2023-01-02) OPT-175B — 8.34 (2023-01-02) SparseGPT (175B, 4:8 Sparsity) — 8.45 (2023-01-02) SparseGPT (175B, 2:4 Sparsity) — 8.73 (2023-01-02) OPT-175B (50% Sparsity) — 234.77 (2023-01-02) Ensemble of All — 53.73 (2023-11-28) Inan et al. (2016) - Variational LSTM (tied) (h=650) + augmented loss — 87.0 (2016-11-04) Grave et al. (2016) - LSTM + continuous cache pointer — 68.9 (2016-12-13) Melis et al. (2017) - 1-layer LSTM (tied) — 65.9 (2017-07-18) AWD-LSTM + continuous cache pointer — 52.0 (2017-08-07) GL-LWGC + AWD-MoS-LSTM + dynamic eval — 40.46 (2017-08-29) Past Decode Reg. + AWD-LSTM-MoS + dyn. eval. — 40.3 (2018-08-14) FRAGE + AWD-LSTM-MoS + dynamic eval — 39.14 (2018-09-18) GPT-2 — 18.34 (2019-02-14) GPT-2 (fine-tuned) — 15.17 (2021-10-16) SparseGPT (175B, 50% Sparsity) — 8.21 (2023-01-02)
RankModel Test perplexityValidation perplexityNumber of params Extra Training Data PaperCodeYear
1 SparseGPT (175B, 50% Sparsity) 8.21–– ✓ SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot nvidia/tensorrt-model-optimizer · ist-daslab/sparsegpt · nvlabs/maskllm · +3 2023
2 OPT-175B 8.34–– ✓ SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot nvidia/tensorrt-model-optimizer · ist-daslab/sparsegpt · nvlabs/maskllm · +3 2023
3 SparseGPT (175B, 4:8 Sparsity) 8.45–– ✓ SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot nvidia/tensorrt-model-optimizer · ist-daslab/sparsegpt · nvlabs/maskllm · +3 2023
4 SparseGPT (175B, 2:4 Sparsity) 8.73–– ✓ SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot nvidia/tensorrt-model-optimizer · ist-daslab/sparsegpt · nvlabs/maskllm · +3 2023
5 GPT-2 (fine-tuned) 15.1715.691542M ✓ Hydra: A System for Large Multi-Model Deep Learning knagrecha/hydra 2021
6 GPT-2 18.34–1542M ✓ Language Models are Unsupervised Multitask Learners huggingface/transformers · openai/gpt-2 · PaddlePaddle/PaddleNLP · +18 2019
7 GPT-2 (large) 19.93–762M ✓ Language Models are Unsupervised Multitask Learners huggingface/transformers · openai/gpt-2 · PaddlePaddle/PaddleNLP · +18 2019
8 GPT-2 (medium) 22.76–345M ✓ Language Models are Unsupervised Multitask Learners huggingface/transformers · openai/gpt-2 · PaddlePaddle/PaddleNLP · +18 2019
9 GPT-2 (small) 29.41–117M ✓ Language Models are Unsupervised Multitask Learners huggingface/transformers · openai/gpt-2 · PaddlePaddle/PaddleNLP · +18 2019
10 BERT-Large-CAS 34.137.7395M ✓ Language Models with Transformers cgraywang/gluon-nlp-1 2019
11 Mogrifier LSTM + dynamic eval 38.640.235M Mogrifier LSTM deepmind/lamb · RMichaelSwan/MogrifierLSTM · microcoder-py/mogrifier-lstm 2019
12 adversarial + AWD-LSTM-MoS + dynamic eval 38.6540.2735M Improving Neural Language Modeling via Adversarial Training ChengyueGongR/advsoft 2019
13 FRAGE + AWD-LSTM-MoS + dynamic eval 39.1440.8535M FRAGE: Frequency-Agnostic Word Representation ChengyueGongR/FrequencyAgnostic · JakubStefko/w2vf 2018
14 Past Decode Reg. + AWD-LSTM-MoS + dyn. eval. 40.342.035M Improved Language Modeling by Decoding the Past 2018
15 GL-LWGC + AWD-MoS-LSTM + dynamic eval 40.4642.1938M Gradual Learning of Recurrent Neural Networks zivaharoni/gradual-learning-rnn 2017
16 AWD-LSTM-MoS + dynamic eval 40.6842.4135M Breaking the Softmax Bottleneck: A High-Rank RNN Language Model zihangdai/mos · cstorm125/thai2fit · tdmeeste/SparseSeqModels · +6 2017
17 AWD-LSTM-DRILL + dynamic eval 42.043.934M Deep Residual Output Layers for Neural Language Generation idiap/drill 2019
18 AWD-LSTM + dynamic eval 44.346.433M Dynamic Evaluation of Neural Sequence Models benkrause/dynamic-evaluation · benkrause/dynamiceval-transformer · sacmehta/PRU 2017
19 AWD-LSTM + continuous cache pointer 52.053.833M Regularizing and Optimizing LSTM Language Models google-research/google-research · fastai/fastai · dmlc/gluon-nlp · +42 2017
20 AWD-LSTM-DOC x5 53.0954.19185M Direct Output Connection for a High-Rank Language Model nttcslab-nlp/doc_lm 2018
1–20 / 38 다음 → 페이지당 10 20 50 100