DialCoT Meets PPO: Decomposing and Exploring Reasoning Paths in Smaller Language Models

2023-10-08 · Chengcheng Han, Xiaowei Du, Che Zhang, Yixin Lian, Xiang Li, Ming Gao, Baoyuan Wang

Chain-of-Thought (CoT) prompting has proven to be effective in enhancing the reasoning capabilities of Large Language Models (LLMs) with at least 100 billion parameters. However, it is ineffective or even detrimental when applied to reasoning tasks in Smaller Language Models (SLMs) with less than 10 billion parameters. To address this limitation, we introduce Dialogue-guided Chain-of-Thought (DialCoT) which employs a dialogue format to generate intermediate reasoning steps, guiding the model toward the final answer. Additionally, we optimize the model's reasoning path selection using the Proximal Policy Optimization (PPO) algorithm, further enhancing its reasoning capabilities. Our method offers several advantages compared to previous approaches. Firstly, we transform the process of solving complex reasoning questions by breaking them down into a series of simpler sub-questions, significantly reducing the task difficulty and making it more suitable for SLMs. Secondly, we optimize the model's reasoning path selection through the PPO algorithm. We conduct comprehensive experiments on four arithmetic reasoning datasets, demonstrating that our method achieves significant performance improvements compared to state-of-the-art competitors.

📄 PDF Abstract BibTeX arXiv:2310.05074

Code (1)

hccngu/dialcot 공식 구현 pytorch

Tasks

Arithmetic Reasoning

Methods 이 논문이 사용한 방법론

Entropy Regularization 설명 없음

PPO Proximal Policy Optimization, or PPO, is a policy gradient method for reinforcement learning. The motivation was to have an algorithm with the data efficiency and reliable…

ACE-RTL: When Agentic Context Evolution Meets RTL-Specialized LLMs

2026-02-10 · Chenhui Deng, Zhongzhi Yu, Guan-Ting Liu, Nathaniel Pinckney 외 arxiv

Recent advances in LLMs have sparked growing interest in applying them to hardware design automation, particularly for accurate RTL code generation. Prior efforts follow two largely independent paths: (i) training domain…

Code Generation

Learning to Walk with Dual Agents for Knowledge Graph Reasoning

2021-12-23 · Denghui Zhang, Zixuan Yuan, Hao liu, Xiaodong Lin 외

Graph walking based on reinforcement learning (RL) has shown great success in navigating an agent to automatically complete various reasoning tasks over an incomplete knowledge graph (KG) by exploring multi-hop relationa…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Improving LLM Reasoning with Multi-Agent Tree-of-Thought Validator Agent

2024-09-17 · Fatemeh Haji, Mazal Bethany, Maryam Tabar, Jason Chiang 외

Multi-agent strategies have emerged as a promising approach to enhance the reasoning abilities of Large Language Models (LLMs) by assigning specialized roles in the problem-solving process. Concurrently, Tree of Thoughts…

GSM8K Question Answering valid

Please Translate Again: Two Simple Experiments on Whether Human-Like Reasoning Helps Translation

2025-06-05 · Di wu, Seth Aycock, Christof Monz

Large Language Models (LLMs) demonstrate strong reasoning capabilities for many tasks, often by explicitly decomposing the task via Chain-of-Thought (CoT) reasoning. Recent work on LLM-based translation designs hand-craf…

Translation

Forest-of-Thought: Scaling Test-Time Compute for Enhancing LLM Reasoning

2024-12-12 · Zhenni Bi, Kai Han, Chuanjian Liu, Yehui Tang 외

Large Language Models (LLMs) have demonstrated remarkable abilities across various language tasks, but solving complex reasoning problems remains a significant challenge. While existing methods, such as Chain-of-Thought …

Decision Making