Coarse-Tuning Models of Code with Reinforcement Learning Feedback
Large Language Models (LLMs) pre-trained on code have recently emerged as the dominant approach to program synthesis. However, these models are trained using next-token prediction, which ignores the syntax and semantics of code. We propose RLCF, that further trains a pre-trained LLM via reinforcement learning, using feedback from a grounding function that scores the quality of the code. The grounding function uses (i) compiler-derived feedback on whether the code it generates passes a set of correctness checks; and (ii) feedback from a different LLM that compares the generated code to a reference code. RLCF is model- and language-agnostic. We empirically evaluate it on the MBJP and MathQA tasks for Java. Our experiments show that RLCF raises the odds that an LLM-generated program compiles, is executable, and produces the right output on tests, often allowing LLMs to match the performance of 2x-8x larger LLMs.
Code (0)
등록된 구현이 없습니다.
Tasks
Program Synthesisreinforcement-learningReinforcement LearningSimilar Papers 제목 키워드 기반
Reinforcement Learning with Token-level Feedback for Controllable Text Generation
To meet the requirements of real-world applications, it is essential to control generations of large language models (LLMs). Prior research has tried to introduce reinforcement learning (RL) into controllable text genera…
Attributereinforcement-learningReinforcement LearningReinforcement Learning (RL)+2BoostAPR: Boosting Automated Program Repair via Execution-Grounded Reinforcement Learning with Dual Reward Models
Reinforcement learning for program repair is hindered by sparse execution feedback and coarse sequence-level rewards that obscure which edits actually fix bugs. We present BoostAPR, a three-stage framework addressing the…
Reinforcement LearningProgram RepairOn-Policy Fine-grained Knowledge Feedback for Hallucination Mitigation
Hallucination occurs when large language models (LLMs) exhibit behavior that deviates from the boundaries of their knowledge during the response generation process. Previous learning-based methods focus on detecting know…
HallucinationResponse GenerationHERO: Human-Feedback Efficient Reinforcement Learning for Online Diffusion Model Finetuning
Controllable generation through Stable Diffusion (SD) fine-tuning aims to improve fidelity, safety, and alignment with human guidance. Existing reinforcement learning from human feedback methods usually rely on predefine…
Image Generationreinforcement-learningReinforcement LearningRepresentation LearningConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs
Reinforcement learning (RL) has become a standard paradigm for refining large language models (LLMs) beyond pre-training and instruction tuning. A prominent line of work is RL with verifiable rewards (RLVR), which levera…
Reinforcement Learning