DisasterQA: A Benchmark for Assessing the performance of LLMs in Disaster Response
Disasters can result in the deaths of many, making quick response times vital. Large Language Models (LLMs) have emerged as valuable in the field. LLMs can be used to process vast amounts of textual information quickly providing situational context during a disaster. However, the question remains whether LLMs should be used for advice and decision making in a disaster. To evaluate the capabilities of LLMs in disaster response knowledge, we introduce a benchmark: DisasterQA created from six online sources. The benchmark covers a wide range of disaster response topics. We evaluated five LLMs each with four different prompting methods on our benchmark, measuring both accuracy and confidence levels through Logprobs. The results indicate that LLMs require improvement on disaster response knowledge. We hope that this benchmark pushes forth further development of LLMs in disaster response, ultimately enabling these models to work alongside. emergency managers in disasters.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingDisaster ResponseSimilar Papers 제목 키워드 기반
Assessing out-of-domain generalization for robust building damage detection
An important step for limiting the negative impact of natural disasters is rapid damage assessment after a disaster occurred. For instance, building damage detection can be automated by applying computer vision technique…
Domain GeneralizationAssessing On-the-Ground Disaster Impact Using Online Data Sources
Assessing the impact of a disaster in terms of asset losses and human casualties is essential for preparing effective response plans. Traditional methods include offline assessments conducted on the ground, where volunte…
DisasterBench: A Multimodal Benchmark for UAV-Based Disaster Response in Complex Environments
When a disaster unfolds, responders must answer not only what is happening, but also why it is happening, what will happen next, and what to do now, often from noisy low-altitude UAV views and under tight on-site compute…
Reinforcement LearningMultimodal ReasoningHarnessing Large Language Models for Disaster Management: A Survey
Large language models (LLMs) have revolutionized scientific research with their exceptional capabilities and transformed various fields. Among their practical applications, LLMs have been playing a crucial role in mitiga…
ManagementSurveyMONITRS: Multimodal Observations of Natural Incidents Through Remote Sensing
Natural disasters cause devastating damage to communities and infrastructure every year. Effective disaster response is hampered by the difficulty of accessing affected areas during and after events. Remote sensing has a…