ElicitationGPT: Text Elicitation Mechanisms via Language Models
Scoring rules evaluate probabilistic forecasts of an unknown state against the realized state and are a fundamental building block in the incentivized elicitation of information and the training of machine learning models. This paper develops mechanisms for scoring elicited text against ground truth text using domain-knowledge-free queries to a large language model (specifically ChatGPT) and empirically evaluates their alignment with human preferences. The empirical evaluation is conducted on peer reviews from a peer-grading dataset and in comparison to manual instructor scores for the peer reviews.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingLarge Language ModelSimilar Papers 제목 키워드 기반
Accelerated Preference Elicitation with LLM-Based Proxies
Bidders in combinatorial auctions face significant challenges when describing their preferences to an auctioneer. Classical work on preference elicitation focuses on query-based techniques inspired from proper learning--…
Is Reasoning All You Need? Probing Bias in the Age of Reasoning Language Models
Reasoning Language Models (RLMs) have gained traction for their ability to perform complex, multi-step reasoning tasks through mechanisms such as Chain-of-Thought (CoT) prompting or fine-tuned reasoning traces. While the…
Adversarial RobustnessAllFairnessIdentification of misreported beliefs
It is well-known that subjective beliefs cannot be identified with traditional choice data unless we impose the strong assumption that preferences are state-independent. This is seen as one of the biggest pitfalls of inc…
Mechanisms for belief elicitation without ground truth
This review article examines the challenge of eliciting truthful information from multiple individuals when such information cannot be verified, a problem known as information elicitation without verification (IEWV). Thi…
Mind the Gap: How Elicitation Protocols Shape the Stated-Revealed Preference Gap in Language Models
Recent work identifies a stated-revealed (SvR) preference gap in language models (LMs): a mismatch between the values models endorse and the choices they make in context. Existing evaluations rely heavily on binary force…