Execution-Based Evaluation for Open-Domain Code Generation

2022-12-20 · Zhiruo Wang, Shuyan Zhou, Daniel Fried, Graham Neubig

To extend the scope of coding queries to more realistic settings, we propose ODEX, the first Open-Domain EXecution-based natural language (NL) to Python code generation dataset. ODEX has 945 NL-Code pairs spanning 79 diverse libraries, along with 1,707 human-written test cases for execution. Our NL-Code pairs are harvested from StackOverflow forums to encourage natural and practical coding queries. Moreover, ODEX supports four natural languages as intents, in English, Spanish, Japanese, and Russian. ODEX unveils intriguing behavioral differences among top-performing code language models (LM). While CODEX achieves better overall results, CODEGEN improves effectively via scaling -- CODEGEN 6.1B performs comparably with CODEX 12B. Both models show substantial gaps between open and closed domains, but CODEGEN gaps tend to decrease with model size while CODEX gaps increase. We release ODEX to facilitate research into open-domain problems for the code generation community.

📄 PDF Abstract BibTeX arXiv:2212.10481

Code (1)

zorazrw/odex 공식 구현

Tasks

Code Generation Memorization

Methods 이 논문이 사용한 방법론

Test 설명 없음

CodeGen CodeGen is an autoregressive transformers with next-token prediction language modeling as the learning objective trained on a natural language corpus and programming language…

BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution

2025-10-09 · Terry Yue Zhuo, Xiaolong Jin, Hange Liu, Juyong Jiang 외 arxiv

Crowdsourced model evaluation platforms, such as Chatbot Arena, enable real-time evaluation from human perspectives to assess the quality of model responses. In the coding domain, manually examining the quality of LLM-ge…

Code Generation

Benchmarks for Trajectory Safety Evaluation and Diagnosis in OpenClaw and Codex: ATBench-Claw and ATBench-Codex

2026-04-16 · Zhonghao Yang, Yu Li, Yanxu Zhu, Tianyi Zhou 외 arxiv

As agent systems move into increasingly diverse execution settings, trajectory-level safety evaluation and diagnosis require benchmarks that evolve with them. ATBench is a diverse and realistic agent trajectory benchmark…

OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement

2024-02-22 · Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu 외

The introduction of large language models has significantly advanced code generation. However, open-source models often lack the execution capabilities and iterative refinement of advanced systems like the GPT-4 Code Int…

Code Generation HumanEval mbpp

Execution-Based Evaluation of Natural Language to Bash and PowerShell for Incident Remediation

2024-05-10 · Ngoc Phuoc An Vo, Brent Paulovicks, Vadim Sheinin

Given recent advancements of Large Language Models (LLMs), code generation tasks attract immense attention for wide application in different domains. In an effort to evaluate and select a best model to automatically reme…

Code Generation Few-Shot Learning

CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarks

2024-03-31 · Yiqing Xie, Alex Xie, Divyanshu Sheth, PengFei Liu 외

To adequately test modern code generation systems, evaluation benchmarks must execute and test the code generated by the system. However, these execution and testing requirements have largely limited benchmarks to settin…

Code Generation Language Modeling Language Modelling Large Language Model