Code Generation 벤치마크
Code Generation on WebApp1K-React
pass@1
- 2024-09-08 — gpt-4o-2024-08-06: pass@1 0.885
- 2024-09-19 — o1-preview: pass@1 0.952
| Rank | Model | pass@1 | Paper | Code | Year |
|---|---|---|---|---|---|
| 1 | o1-preview | 0.952 | A Case Study of Web App Coding with OpenAI Reasoning Models | onekq/webapp1k | 2024 |
| 2 | o1-mini | 0.939 | A Case Study of Web App Coding with OpenAI Reasoning Models | onekq/webapp1k | 2024 |
| 3 | gpt-4o-2024-08-06 | 0.885 | Insights from Benchmarking Frontier Language Models on Web App Code Generation | onekq/webapp1k | 2024 |
| 4 | claude-3.5-sonnet | 0.8808 | Insights from Benchmarking Frontier Language Models on Web App Code Generation | onekq/webapp1k | 2024 |
| 5 | deepseek-v2.5 | 0.834 | A Case Study of Web App Coding with OpenAI Reasoning Models | onekq/webapp1k | 2024 |
| 6 | mistral-large-2 | 0.7804 | Insights from Benchmarking Frontier Language Models on Web App Code Generation | onekq/webapp1k | 2024 |
| 7 | deepseek-coder-v2-instruct | 0.7002 | Insights from Benchmarking Frontier Language Models on Web App Code Generation | onekq/webapp1k | 2024 |
| 8 | llama-v3p1-405b-instruct | 0.302 | Insights from Benchmarking Frontier Language Models on Web App Code Generation | onekq/webapp1k | 2024 |