Ensembling Large Language Models with Process Reward-Guided Tree Search for Better Complex Reasoning

2024-12-20 · Sungjin Park, Xiao Liu, Yeyun Gong, Edward Choi

Despite recent advances in large language models, open-source models often struggle to consistently perform well on complex reasoning tasks. Existing ensemble methods, whether applied at the token or output levels, fail to address these challenges. In response, we present Language model Ensemble with Monte Carlo Tree Search (LE-MCTS), a novel framework for process-level ensembling of language models. LE-MCTS formulates step-by-step reasoning with an ensemble of language models as a Markov decision process. In this framework, states represent intermediate reasoning paths, while actions consist of generating the next reasoning step using one of the language models selected from a predefined pool. Guided by a process-based reward model, LE-MCTS performs a tree search over the reasoning steps generated by different language models, identifying the most accurate reasoning chain. Experimental results on five mathematical reasoning benchmarks demonstrate that our approach outperforms both single language model decoding algorithms and language model ensemble methods. Notably, LE-MCTS improves performance by 3.6% and 4.3% on the MATH and MQA datasets, respectively, highlighting its effectiveness in solving complex reasoning problems.

📄 PDF Abstract BibTeX arXiv:2412.15797

Code (0)

등록된 구현이 없습니다.

Tasks

Language ModelingLanguage ModellingMathMathematical Reasoning

TEGEE: Task dEfinition Guided Expert Ensembling for Generalizable and Few-shot Learning

2024-03-07 · Xingwei Qu, Yiming Liang, Yucheng Wang, Tianyu Zheng 외

Large Language Models (LLMs) exhibit the ability to perform in-context learning (ICL), where they acquire new tasks directly from examples provided in demonstrations. This process is thought to operate through an implici…

Continual Learning Definition Extraction Few-Shot Learning In-Context Learning+1

Simple is Better and Large is Not Enough: Towards Ensembling of Foundational Language Models

2023-08-23 · Nancy Tyagi, Aidin Shiri, Surjodeep Sarkar, Abhishek Kumar Umrawal 외

Foundational Language Models (FLMs) have advanced natural language processing (NLP) research. Current researchers are developing larger FLMs (e.g., XLNet, T5) to enable contextualized language representation, classificat…

Hallucination

FUSE: Ensembling Verifiers with Zero Labeled Data

2026-04-20 · Joonhyuk Lee, Virginia Ma, Sarah Zhao, Yash Nair 외 arxiv

Verification of model outputs is rapidly emerging as a key primitive for both training and real-world deployment of large language models (LLMs). In practice, this often involves using imperfect LLM judges and reward mod…

Leveraging Knowledge and Reinforcement Learning for Enhanced Reliability of Language Models

2023-08-25 · Nancy Tyagi, Surjodeep Sarkar, Manas Gaur

The Natural Language Processing(NLP) community has been using crowd sourcing techniques to create benchmark datasets such as General Language Understanding and Evaluation(GLUE) for training modern Language Models such as…

Knowledge Graph Embeddings reinforcement-learning

WARM: On the Benefits of Weight Averaged Reward Models

2024-01-22 · Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi 외

Aligning large language models (LLMs) with human preferences through reinforcement learning (RLHF) can lead to reward hacking, where LLMs exploit failures in the reward model (RM) to achieve seemingly high rewards withou…