Stereotypical Bias Analysis 벤치마크
Stereotypical Bias Analysis on CrowS-Pairs
Gender
- 2022-05-02 — GPT-3: Gender 62.6
- 2022-05-02 — OPT-175B: Gender 65.7
- 2023-02-27 — LLaMA 65B: Gender 70.6
| Rank | Model | Gender | Religion | Race/Color | Sexual Orientation | Age | Nationality | Disability | Physical Appearance | Paper | Code | Year |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | GAL 120B | 51.9 | 51.9 | 59.9 | 77.4 | 69 | 51.6 | 66.7 | 58.7 | Galactica: A Large Language Model for Science | paperswithcode/galai | 2022 |
| 2 | GPT-3 | 62.6 | 62.6 | 64.7 | 76.2 | 64.4 | 61.6 | 76.7 | 74.6 | OPT: Open Pre-trained Transformer Language Models | facebookresearch/metaseq · pku-alignment/safe-rlhf · liangyuwang/zo2 · +8 | 2022 |
| 3 | OPT-175B | 65.7 | 65.7 | 68.6 | 78.6 | 67.8 | 62.9 | 76.7 | 76.2 | OPT: Open Pre-trained Transformer Language Models | facebookresearch/metaseq · pku-alignment/safe-rlhf · liangyuwang/zo2 · +8 | 2022 |
| 4 | LLaMA 65B | 70.6 | 70.6 | 57.0 | 81.0 | 70.1 | 64.2 | 66.7 | 77.8 | LLaMA: Open and Efficient Foundation Language Models | huggingface/transformers · ggml-org/llama.cpp · ggerganov/llama.cpp · +54 | 2023 |