Language Modelling 벤치마크
Language Modelling on SALMon
Sentiment Consistency 낮을수록 좋음
- 2021-09-07 — pGSLM: Sentiment Consistency 40.5
| Rank | Model | Sentiment Consistency | Speaker Consistency | Gender Consistency | Background (Domain) Consistency | Background (Random) Consistency | Room Consistency | Sentiment Alignment | Background Alignment | Extra Training Data | Paper | Code | Year |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Spirit-LM (Expr.) | 73.5 | 81.0 | 85.0 | 55.0 | 64.0 | 54.5 | 52.0 | 59.5 | ✓ | Spirit LM: Interleaved Spoken and Written Language Model | facebookresearch/spiritlm | 2024 |
| 2 | LAST 1.3B | 65.0 | 64.5 | 68.5 | 56.0 | 61.0 | 62.5 | 53.5 | 53.0 | ✓ | LAST: Language Model Aware Speech Tokenization | 2024 | |
| 3 | LAST 350M | 64.0 | 63.0 | 70.5 | 55.5 | 60.5 | 61.0 | 51.5 | 54.5 | ✓ | LAST: Language Model Aware Speech Tokenization | 2024 | |
| 4 | TWIST 7B | 61.5 | 71.0 | 70.0 | 55.0 | 60.5 | 62.0 | 51.5 | 54.5 | ✓ | Textually Pretrained Speech Language Models | slp-rl/spokenstorycloze | 2023 |
| 4 | TWIST 1.3B | 61.5 | 69.0 | 69.5 | 55.5 | 60.5 | 59.0 | 53.0 | 56.5 | ✓ | Textually Pretrained Speech Language Models | slp-rl/spokenstorycloze | 2023 |
| 6 | TWIST 350M | 59.0 | 69.5 | 68.0 | 54.0 | 61.5 | 59.0 | 51.5 | 56.5 | ✓ | Textually Pretrained Speech Language Models | slp-rl/spokenstorycloze | 2023 |
| 6 | TASLM 1B (token) | 59.0 | 68.0 | 70.5 | – | – | 61.0 | – | – | ||||
| 8 | TASLM 1B (embedding) | 57.5 | 67.0 | 75.5 | – | – | 50.0 | – | – | ||||
| 9 | Spirit-LM (base) | 54.5 | 69.5 | 67.0 | 53.5 | 55.5 | 54.5 | 48.0 | 51.5 | ✓ | Spirit LM: Interleaved Spoken and Written Language Model | facebookresearch/spiritlm | 2024 |
| 10 | pGSLM | 40.5 | 83.0 | 88.5 | 57.0 | 66.0 | 53.5 | 55.5 | 53.5 | ✓ | Text-Free Prosody-Aware Generative Spoken Language Modeling | pytorch/fairseq | 2021 |