ClimateX: Do LLMs Accurately Assess Human Expert Confidence in Climate Statements?
Evaluating the accuracy of outputs generated by Large Language Models (LLMs) is especially important in the climate science and policy domain. We introduce the Expert Confidence in Climate Statements (ClimateX) dataset, a novel, curated, expert-labeled dataset consisting of 8094 climate statements collected from the latest Intergovernmental Panel on Climate Change (IPCC) reports, labeled with their associated confidence levels. Using this dataset, we show that recent LLMs can classify human expert confidence in climate-related statements, especially in a few-shot learning setting, but with limited (up to 47%) accuracy. Overall, models exhibit consistent and significant over-confidence on low and medium confidence statements. We highlight implications of our results for climate communication, LLMs evaluation strategies, and the use of LLMs in information retrieval systems.
Code (1)
Tasks
Few-Shot LearningInformation RetrievalRetrievalSimilar Papers 제목 키워드 기반
Don't Think Twice! Over-Reasoning Impairs Confidence Calibration
Large Language Models deployed as question answering tools require robust calibration to avoid overconfidence. We systematically evaluate how reasoning capabilities and budget affect confidence assessment accuracy, using…
Question AnsweringFine-tuning ClimateBert transformer with ClimaText for the disclosure analysis of climate-related financial risks
In recent years there has been a growing demand from financial agents, especially from particular and institutional investors, for companies to report on climate-related financial risks. A vast amount of information, in …
text-classificationText ClassificationTransfer LearningInterpretable Humans, Alien LLMs: Expert Analysis of Latent Structures in Assessment Responses
The evaluation of large language models (LLMs) relies heavily on human-designed assessments, implicitly assuming that AI and humans employ similar underlying cognitive constructs. Challenging this assumption, we investig…
CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge
Large Language Models (LLMs) are increasingly used across various domains, from software development to cyber threat intelligence. Understanding all the different fields of cybersecurity, which includes topics such as cr…
General KnowledgeMultiple-choiceRAGRetrieval+1A Computational Framework for Behavioral Assessment of LLM Therapists
The emergence of large language models (LLMs) like ChatGPT has increased interest in their use as therapists to address mental health challenges and the widespread lack of access to care. However, experts have emphasized…
In-Context Learning