paper-with-me

Code Generation 벤치마크

Code Generation on RES-Q

9개 결과 · ⬇ CSV · JSON

pass@1

18 28 38 48 58 2024-06 2026-11 QurrentOS-coder + Claude 3.5 Sonnet — 58.0 (2024-06-24) QurrentOS-coder + GPT-4o — 46.0 (2024-06-24) QurrentOS-coder + GPT-4 Turbo — 37.0 (2024-06-24) QurrentOS-coder + Claude 3 Opus — 36.0 (2024-06-24) QurrentOS-coder + GPT-4 — 30.0 (2024-06-24) QurrentOS-coder + Gemini 1.5 Pro — 30.0 (2024-06-24) QurrentOS-coder + DeepSeek-Coder-V2 — 29.0 (2024-06-24) QurrentOS-coder + Llama 3 70b — 20.0 (2024-06-24) QurrentOS-coder + Qwen-72B-Instruct — 18.0 (2024-06-24) QurrentOS-coder + Claude 3.5 Sonnet — 58.0 (2024-06-24)
RankModel pass@1 PaperCodeYear
1 QurrentOS-coder + Claude 3.5 Sonnet 58.0 RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale qurrent-ai/res-q 2024
2 QurrentOS-coder + GPT-4o 46.0 RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale qurrent-ai/res-q 2024
3 QurrentOS-coder + GPT-4 Turbo 37.0 RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale qurrent-ai/res-q 2024
4 QurrentOS-coder + Claude 3 Opus 36.0 RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale qurrent-ai/res-q 2024
5 QurrentOS-coder + GPT-4 30.0 RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale qurrent-ai/res-q 2024
5 QurrentOS-coder + Gemini 1.5 Pro 30.0 RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale qurrent-ai/res-q 2024
7 QurrentOS-coder + DeepSeek-Coder-V2 29.0 RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale qurrent-ai/res-q 2024
8 QurrentOS-coder + Llama 3 70b 20.0 RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale qurrent-ai/res-q 2024
9 QurrentOS-coder + Qwen-72B-Instruct 18.0 RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale qurrent-ai/res-q 2024
1–9 / 9 페이지당 10 20 50 100