AutoHall: Automated Hallucination Dataset Generation for Large Language Models
While Large language models (LLMs) have garnered widespread applications across various domains due to their powerful language understanding and generation capabilities, the detection of non-factual or hallucinatory content generated by LLMs remains scarce. Currently, one significant challenge in hallucination detection is the laborious task of time-consuming and expensive manual annotation of the hallucinatory generation. To address this issue, this paper first introduces a method for automatically constructing model-specific hallucination datasets based on existing fact-checking datasets called AutoHall. Furthermore, we propose a zero-resource and black-box hallucination detection method based on self-contradiction. We conduct experiments towards prevalent open-/closed-source LLMs, achieving superior hallucination detection performance compared to extant baselines. Moreover, our experiments reveal variations in hallucination proportions and types among different models.
Code (0)
등록된 구현이 없습니다.
Tasks
Dataset GenerationFact CheckingHallucinationSimilar Papers 제목 키워드 기반
AutoHallusion: Automatic Generation of Hallucination Benchmarks for Vision-Language Models
Large vision-language models (LVLMs) are prone to hallucinations, where certain contextual cues in an image can trigger the language module to produce overconfident and incorrect reasoning about abnormal or hypothetical …
HallucinationHallucination EvaluationVisual Question Answering (VQA)DAHL: Domain-specific Automated Hallucination Evaluation of Long-Form Text through a Benchmark Dataset in Biomedicine
We introduce DAHL, a benchmark dataset and automated evaluation system designed to assess hallucination in long-form text generation, specifically within the biomedical domain. Our benchmark dataset, meticulously curated…
FormHallucinationHallucination EvaluationMultiple-choice+1HalluScope: Fine-grained Hallucination Diagnosis for Multimodal Large Language Models
Although Multimodal Large Language Models have achieved strong performance across a wide range of vision-language tasks, they still suffer from hallucinations, where model outputs become inconsistent with the visual cont…
Explanation GenerationTask Abstention for Large Language Models in Code Generation
Large language models (LLMs) have revolutionized automated code generation. One serious concern, however, is the so-called ``hallucination'', i.e., LLMs may generate seemingly plausible but functionally incorrect code. I…
Code GenerationBetter Understanding, Better Fixes? A Study of Hallucination in LLM-based Automated Program Repair
Large language models (LLMs) have significantly advanced automated program repair (APR), yet existing evaluations remain largely result-centric and provide limited insight into hallucination during repair. In APR, halluc…
Program Repair