Shepherd: A Critic for Language Model Generation
As large language models improve, there is increasing interest in techniques that leverage these models' capabilities to refine their own outputs. In this work, we introduce Shepherd, a language model specifically tuned to critique responses and suggest refinements, extending beyond the capabilities of an untuned model to identify diverse errors and provide suggestions to remedy them. At the core of our approach is a high quality feedback dataset, which we curate from community feedback and human annotations. Even though Shepherd is small (7B parameters), its critiques are either equivalent or preferred to those from established models including ChatGPT. Using GPT-4 for evaluation, Shepherd reaches an average win-rate of 53-87% compared to competitive alternatives. In human evaluation, Shepherd strictly outperforms other models and on average closely ties with ChatGPT.
Code (1)
Tasks
Language ModelingLanguage ModellingmodelMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
VIS-Shepherd: Constructing Critic for LLM-based Data Visualization Generation
Data visualization generation using Large Language Models (LLMs) has shown promising results but often produces suboptimal visualizations that require human intervention for improvement. In this work, we introduce VIS-Sh…
Data VisualizationLanguage ModelingLanguage ModellingLarge Language Model+1Pay for Hints, Not Answers: LLM Shepherding for Cost-Efficient Inference
Large Language Models (LLMs) deliver state-of-the-art performance on complex reasoning tasks, but their inference costs limit deployment at scale. Small Language Models (SLMs) offer dramatic cost savings yet lag substant…
Mathematical ReasoningCode GenerationMath-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations
In this paper, we present an innovative process-oriented math process reward model called \textbf{Math-Shepherd}, which assigns a reward score to each step of math problem solutions. The training of Math-Shepherd is achi…
Arithmetic ReasoningGSM8KMathMathematical Reasoning+2GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Tasks
Autonomous agents for long-sequence Graphical User Interface tasks are hindered by sparse rewards and the intractable credit assignment problem. To address these challenges, we introduce GUI-Shepherd, a Process Reward Mo…