Responsible Retrieval Augmented Generation for Climate Decision Making from Documents
Climate decision making is constrained by the complexity and inaccessibility of key information within lengthy, technical, and multi-lingual documents. Generative AI technologies offer a promising route for improving the accessibility of information contained within these documents, but suffer from limitations. These include (1) a tendency to hallucinate or mis-represent information, (2) difficulty in steering or guaranteeing properties of generated output, and (3) reduced performance in specific technical domains. To address these challenges, we introduce a novel evaluation framework with domain-specific dimensions tailored for climate-related documents. We then apply this framework to evaluate Retrieval-Augmented Generation (RAG) approaches and assess retrieval- and generation-quality within a prototype tool that answers questions about individual climate law and policy documents. In addition, we publish a human-annotated dataset and scalable automated evaluation tools, with the aim of facilitating broader adoption and robust assessment of these systems in the climate domain. Our findings highlight the key components of responsible deployment of RAG to enhance decision-making, while also providing insights into user experience (UX) considerations for safely deploying such systems to build trust with users in high-risk domains.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingRAGRetrievalRetrieval-augmented GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Listen to the Context: Towards Faithful Large Language Models for Retrieval Augmented Generation on Climate Questions
Large language models that use retrieval augmented generation have the potential to unlock valuable knowledge for researchers, policymakers, and the public by making long and technical climate-related documents more acce…
Language ModelingLanguage ModellingLarge Language ModelRetrieval-augmented GenerationClimRetrieve: A Benchmarking Dataset for Information Retrieval from Corporate Climate Disclosures
To handle the vast amounts of qualitative data produced in corporate climate communication, stakeholders increasingly rely on Retrieval Augmented Generation (RAG) systems. However, a significant gap remains in evaluating…
Answer GenerationBenchmarkingInformation RetrievalRAG+2VERA: Validation and Evaluation of Retrieval-Augmented Systems
The increasing use of Retrieval-Augmented Generation (RAG) systems in various applications necessitates stringent protocols to ensure RAG systems accuracy, safety, and alignment with user intentions. In this paper, we in…
Decision MakingRAGRetrievalRetrieval-augmented GenerationClimate Finance Bench
Climate Finance Bench introduces an open benchmark that targets question-answering over corporate climate disclosures using Large Language Models. We curate 33 recent sustainability reports in English drawn from companie…
Logical ReasoningQuantizationQuestion AnsweringRAG+2Unbox Responsible GeoAI: Navigating Climate Extreme and Disaster Mapping
As climate extreme and disaster events become more frequent and intense, Geospatial Artificial Intelligence (GeoAI) has emerged as a transformative approach for large-scale disaster mapping and risk reduction. However, t…