Execution-based Code Generation using Deep Reinforcement Learning

2023-01-31 · Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, Chandan K. Reddy

The utilization of programming language (PL) models, pre-trained on large-scale code corpora, as a means of automating software engineering processes has demonstrated considerable potential in streamlining various code generation tasks such as code completion, code translation, and program synthesis. However, current approaches mainly rely on supervised fine-tuning objectives borrowed from text generation, neglecting unique sequence-level characteristics of code, including but not limited to compilability as well as syntactic and functional correctness. To address this limitation, we propose PPOCoder, a new framework for code generation that synergistically combines pre-trained PL models with Proximal Policy Optimization (PPO) which is a widely used deep reinforcement learning technique. By utilizing non-differentiable feedback from code execution and structure alignment, PPOCoder seamlessly integrates external code-specific knowledge into the model optimization process. It's important to note that PPOCoder is a task-agnostic and model-agnostic framework that can be used across different code generation tasks and PLs. Extensive experiments on three code generation tasks demonstrate the effectiveness of our proposed approach compared to SOTA methods, achieving significant improvements in compilation success rates and functional correctness across different PLs.

📄 PDF Abstract BibTeX arXiv:2301.13816

Code (1)

reddy-lab-code-research/PPOCoder 공식 구현 pytorch

Tasks

Code Completion Code Generation Code TranslationDeep Reinforcement LearningModel OptimizationProgram Synthesis reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Text Generation

CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignment

2025-10-21 · Xue Jiang, Yihong Dong, Mengyang Liu, Hongyi Deng 외 arxiv

While Large Language Models (LLMs) excel at code generation by learning from vast code corpora, a fundamental semantic gap remains between their training on textual patterns and the goal of functional correctness, which …

Reinforcement Learning Code Generation

Domain-Adaptable Reinforcement Learning for Code Generation with Dense Rewards

2026-05-20 · Erfan Aghadavoodi Jolfaei, Daniel Maninger, Abhinav Anand, Mert Tiftikci 외 arxiv

Large language models show strong potential for automated code generation, but lack guarantees for correctness, quality, safety, and domain-specific constraints. For instance in robotics, where code generation is increas…

Reinforcement Learning Program Synthesis Code Generation

StepCodeReasoner: Aligning Code Reasoning with Stepwise Execution Traces via Reinforcement Learning

2026-05-12 · Hao Wang, Rui Li, Lei Sha, Jie M. Zhang arxiv

Existing code reasoning methods primarily supervise final code outputs, ignoring intermediate states, often leading to reward hacking where correct answers are obtained through inconsistent reasoning. We propose StepCode…

Reinforcement Learning Code Generation

Process-Supervised Reinforcement Learning for Code Generation

2025-02-03 · Yufan Ye, Ting Zhang, Wenbin Jiang, Hua Huang

Existing reinforcement learning strategies based on outcome supervision have proven effective in enhancing the performance of large language models(LLMs) for code generation. While reinforcement learning based on process…

Code Generation reinforcement-learning Reinforcement Learning

Multi-Turn Code Generation Through Single-Step Rewards

2025-02-27 · Arnav Kumar Jain, Gonzalo Gonzalez-Pumariega, Wayne Chen, Alexander M Rush 외

We address the problem of code generation from multi-turn execution feedback. Existing methods either generate code without feedback or use complex, hierarchical reinforcement learning to optimize multi-turn rewards. We …

Code Generation Hierarchical Reinforcement Learning