LM-Polygraph: Uncertainty Estimation for Language Models
Recent advancements in the capabilities of large language models (LLMs) have paved the way for a myriad of groundbreaking applications in various fields. However, a significant challenge arises as these models often "hallucinate", i.e., fabricate facts without providing users an apparent means to discern the veracity of their statements. Uncertainty estimation (UE) methods are one path to safer, more responsible, and more effective use of LLMs. However, to date, research on UE methods for LLMs has been focused primarily on theoretical rather than engineering contributions. In this work, we tackle this issue by introducing LM-Polygraph, a framework with implementations of a battery of state-of-the-art UE methods for LLMs in text generation tasks, with unified program interfaces in Python. Additionally, it introduces an extendable benchmark for consistent evaluation of UE techniques by researchers, and a demo web application that enriches the standard chat dialog with confidence scores, empowering end-users to discern unreliable responses. LM-Polygraph is compatible with the most recent LLMs, including BLOOMz, LLaMA-2, ChatGPT, and GPT-4, and is designed to support future releases of similarly-styled LMs.
Code (0)
등록된 구현이 없습니다.
Tasks
Text GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph
The rapid proliferation of large language models (LLMs) has stimulated researchers to seek effective and efficient approaches to deal with LLM hallucinations and low-quality outputs. Uncertainty quantification (UQ) is a …
BenchmarkingText GenerationUncertainty QuantificationReal-time Mode-Aware Dataflow: A Dataflow Model to Specify and Analyze Mode-dependent CPSs under Relaxed Timing Constraints
Modern Cyber-Physical Systems (CPS) often exhibit both relaxed real-time constraints and a mode-dependent execution. Relaxed real-time constraints mean that only a subset of the processes of a CPS have real-time constrai…
IngenuityPOLygraph: Polish Fake News Dataset
This paper presents the POLygraph dataset, a unique resource for fake news detection in Polish. The dataset, created by an interdisciplinary team, is composed of two parts: the "fake-or-not" dataset with 11,360 pairs of …
ArticlesFact CheckingFake News DetectionPolyGraph Discrepancy: a classifier-based metric for graph generation
Existing methods for evaluating graph generative models primarily rely on Maximum Mean Discrepancy (MMD) metrics based on graph descriptors. While these metrics can rank generative models, they do not provide an absolute…
Graph Generation