Efficient Post-training of LLMs for Code Generation With Offline Reinforcement Learning
Post-training using online reinforcement learning (RL) is an important training step for LLMs, including code-generating models. However, online RL for code generation involves LLM inference and verification of the generated output, which can take considerable time and resources. In this paper, we explore the application of offline RL to code-generating models by leveraging existing code datasets. Our experiments demonstrate that offline RL is an effective training strategy for improving LLM performance. We show that offline RL can be especially beneficial for small LLMs and challenging coding problems.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement LearningCode GenerationOffline RLSimilar Papers 제목 키워드 기반
Future Policy Approximation for Offline Reinforcement Learning in LLM Reasoning
Reinforcement learning (RL) has emerged as a key driver of post-training for complex reasoning in large language models (LLMs), yet online RL introduces substantial instability and computational overhead. Offline RL offe…
Reinforcement LearningMathematical ReasoningOffline RLBridging Online and Offline RL: Contextual Bandit Learning for Multi-Turn Code Generation
Recently, there have been significant research interests in training large language models (LLMs) with reinforcement learning (RL) on real-world tasks, such as multi-turn code generation. While online RL tends to perform…
Reinforcement LearningCode GenerationOffline RLLanguage Models Can Learn from Verbal Feedback Without Scalar Rewards
LLMs are often trained with RL from human or AI feedback, yet such methods typically compress nuanced feedback into scalar rewards, discarding much of their richness and inducing scale imbalance. We propose treating verb…
Text-to-Image GenerationLightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation
On-policy distillation (OPD) is an effective post-training paradigm for large language models but requires a live teacher server throughout training, resulting in substantial infrastructure overhead. We investigate wheth…
Code GenerationHCAG: Hierarchical Abstraction and Retrieval-Augmented Generation on Theoretical Repositories with LLMs
Existing Retrieval-Augmented Generation (RAG) methods for code struggle to capture the high-level architectural patterns and cross-file dependencies inherent in complex, theory-driven codebases, such as those in algorith…
Code Generation