Code Generation 벤치마크
Code Generation on PECC
Pass@3
- 2024-04-29 — Claude 3 Haiku: Pass@3 27.67
| Rank | Model | Pass@3 | Paper | Code | Year |
|---|---|---|---|---|---|
| 1 | Claude 3 Haiku | 27.67 | PECC: Problem Extraction and Coding Challenges | hallerpatrick/pecc | 2024 |
| 2 | GPT-3.5 Turbo | 23.75 | PECC: Problem Extraction and Coding Challenges | hallerpatrick/pecc | 2024 |
| 3 | codechat-bison | 11.39 | PECC: Problem Extraction and Coding Challenges | hallerpatrick/pecc | 2024 |
| 4 | chat-bison | 8.48 | PECC: Problem Extraction and Coding Challenges | hallerpatrick/pecc | 2024 |
| 5 | Mixtral-8x7B-Instruct | 8.35 | PECC: Problem Extraction and Coding Challenges | hallerpatrick/pecc | 2024 |
| 6 | Phi-3-mini-128k-instruct | 7.18 | PECC: Problem Extraction and Coding Challenges | hallerpatrick/pecc | 2024 |
| 7 | WizardLM-2-7B | 3.72 | PECC: Problem Extraction and Coding Challenges | hallerpatrick/pecc | 2024 |
| 8 | Llama-3-8B-Instruct | 3.1 | PECC: Problem Extraction and Coding Challenges | hallerpatrick/pecc | 2024 |