LLM Benchmark-User Need Misalignment for Climate Change
Climate change is a major socio-scientific issue shapes public decision-making and policy discussions. As large language models (LLMs) increasingly serve as an interface for accessing climate knowledge, whether existing benchmarks reflect user needs is critical for evaluating LLM in real-world settings. We propose a Proactive Knowledge Behaviors Framework that captures the different human-human and human-AI knowledge seeking and provision behaviors. We further develop a Topic-Intent-Form taxonomy and apply it to analyze climate-related data representing different knowledge behaviors. Our results reveal a substantial mismatch between current benchmarks and real-world user needs, while knowledge interaction patterns between humans and LLMs closely resemble those in human-human interactions. These findings provide actionable guidance for benchmark design, RAG system development, and LLM training. Code is available at https://github.com/OuchengLiu/LLM-Misalign-Climate-Change.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Understanding Opinions Towards Climate Change on Social Media
Social media platforms such as Twitter (now known as X) have revolutionized how the public engage with important societal and political topics. Recently, climate change discussions on social media became a catalyst for p…
Community DetectionMisinformationSentiment AnalysisClimateSet: A Large-Scale Climate Model Dataset for Machine Learning
Climate models have been key for assessing the impact of climate change and simulating future climate scenarios. The machine learning (ML) community has taken an increased interest in supporting climate scientists' effor…
Climate ProjectionLearning Twitter User Sentiments on Climate Change with Limited Labeled Data
While it is well-documented that climate change accepters and deniers have become increasingly polarized in the United States over time, there has been no large-scale examination of whether these individuals are prone to…
CLINB: A Climate Intelligence Benchmark for Foundational Models
Evaluating how Large Language Models (LLMs) handle complex, specialized knowledge remains a critical challenge. We address this through the lens of climate change by introducing CLINB, a benchmark that assesses models on…
Question AnsweringWhat shapes climate change perceptions in Africa? A random forest approach
Climate change perceptions are fundamental for adaptation and environmental policy support. Although Africa is one of the most vulnerable regions to climate change, little research has focused on how climate change is pe…