RL-STaR: Theoretical Analysis of Reinforcement Learning Frameworks for Self-Taught Reasoner

2024-10-31 · Fu-Chieh Chang, Yu-Ting Lee, Hui-Ying Shih, Pei-Yuan Wu

The reasoning abilities of large language models (LLMs) have improved with chain-of-thought (CoT) prompting, allowing models to solve complex tasks in a stepwise manner. However, training CoT capabilities requires detailed reasoning data, which is often scarce. The self-taught reasoner (STaR) framework addresses this by using reinforcement learning to automatically generate reasoning steps, reducing reliance on human-labeled data. Although STaR and its variants have demonstrated empirical success, a theoretical foundation explaining these improvements is lacking. This work provides a theoretical framework for understanding the effectiveness of reinforcement learning on CoT reasoning and STaR. Our contributions are: (1) an analysis of policy improvement, showing why LLM reasoning improves iteratively with STaR; (2) conditions for convergence to an optimal reasoning policy; (3) an examination of STaR's robustness, explaining how it can improve reasoning even when incorporating occasional incorrect steps; and (4) criteria for the quality of pre-trained models necessary to initiate effective reasoning improvement. This framework aims to bridge empirical findings with theoretical insights, advancing reinforcement learning approaches for reasoning in LLMs.

📄 PDF Abstract BibTeX arXiv:2410.23912

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning

Reinforcement Learning with Quasi-Hyperbolic Discounting

2024-09-16 · S. R. Eshwar, Mayank Motwani, Nibedita Roy, Gugan Thoppe

Reinforcement learning has traditionally been studied with exponential discounting or the average reward setup, mainly due to their mathematical tractability. However, such frameworks fall short of accurately capturing h…

reinforcement-learning Reinforcement Learning

Bridging State and History Representations: Understanding Self-Predictive RL

2024-01-17 · Tianwei Ni, Benjamin Eysenbach, Erfan Seyedsalehi, Michel Ma 외

Representations are at the core of all deep reinforcement learning (RL) methods for both Markov decision processes (MDPs) and partially observable Markov decision processes (POMDPs). Many representation learning methods …

Deep Reinforcement Learning Reinforcement Learning (RL)Representation Learning

Understanding the Robustness of Distributed Self-Supervised Learning Frameworks Against Non-IID Data

2026-07-02 · Xuanyu Chen, Nan Yang, Shuai Wang, Dong Yuan arxiv

Recent research has introduced distributed self-supervised learning (D-SSL) approaches to leverage vast amounts of unlabeled decentralized data. However, D-SSL faces the critical challenge of data heterogeneity, and ther…

Self-Supervised Learning Contrastive Learning Federated Learning

Truthful Self-Play

2021-06-06 · Shohei Ohsawa

We present a general framework for evolutionary learning to emergent unbiased state representation without any supervision. Evolutionary frameworks such as self-play converge to bad local optima in case of multi-agent re…

Multi-agent Reinforcement Learning Starcraft

A Comparison of Self-Play Algorithms Under a Generalized Framework

2020-06-08 · Daniel Hernandez, Kevin Denamganai, Sam Devlin, Spyridon Samothrakis 외

Throughout scientific history, overarching theoretical frameworks have allowed researchers to grow beyond personal intuitions and culturally biased theories. They allow to verify and replicate existing findings, and to l…

Reinforcement Learning (RL)