Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
We introduce Inference-Time Intervention (ITI), a technique designed to enhance the "truthfulness" of large language models (LLMs). ITI operates by shifting model activations during inference, following a set of directions across a limited number of attention heads. This intervention significantly improves the performance of LLaMA models on the TruthfulQA benchmark. On an instruction-finetuned LLaMA called Alpaca, ITI improves its truthfulness from 32.5% to 65.1%. We identify a tradeoff between truthfulness and helpfulness and demonstrate how to balance it by tuning the intervention strength. ITI is minimally invasive and computationally inexpensive. Moreover, the technique is data efficient: while approaches like RLHF require extensive annotations, ITI locates truthful directions using only few hundred examples. Our findings suggest that LLMs may have an internal representation of the likelihood of something being true, even as they produce falsehoods on the surface.
Code (1)
Tasks
Language ModelingLanguage ModellingTruthfulQASimilar Papers 제목 키워드 기반
Learning to Incentivize: Eliciting Effort via Output Agreement
In crowdsourcing when there is a lack of verification for contributed answers, output agreement mechanisms are often used to incentivize participants to provide truthful answers when the correct answer is hold by the maj…
Non-Linear Inference Time Intervention: Improving LLM Truthfulness
In this work, we explore LLM's internal representation space to identify attention heads that contain the most truthful and accurate information. We further developed the Inference Time Intervention (ITI) framework, whic…
Large Language ModelMultiple-choiceQuestion AnsweringTruthfulQAToken-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Models
Uncertainty quantification (UQ) is a prominent approach for eliciting truthful answers from large language models (LLMs). To date, information-based and consistency-based UQ have been the dominant UQ methods for text gen…
Fact Checkingtext-classificationText ClassificationText Generation+1Strategyproof Mean Estimation from Multiple-Choice Questions
Given n values possessed by n agents, we study the problem of estimating the mean by truthfully eliciting agents' answers to multiple-choice questions about their values. We consider two natural candidates for estimation…
Multiple-choiceEnhanced Language Model Truthfulness with Learnable Intervention and Uncertainty Expression
Large language models (LLMs) can generate long-form and coherent text, yet they often hallucinate facts, which undermines their reliability. To mitigate this issue, inference-time methods steer LLM representations toward…
Language ModelingLanguage ModellingQuestion Answering