REGen: A Reliable Evaluation Framework for Generative Event Argument Extraction
Event argument extraction identifies arguments for predefined event roles in text. Traditional evaluations rely on exact match (EM), requiring predicted arguments to match annotated spans exactly. However, this approach fails for generative models like large language models (LLMs), which produce diverse yet semantically accurate responses. EM underestimates performance by disregarding valid variations, implicit arguments (unstated but inferable), and scattered arguments (distributed across a document). To bridge this gap, we introduce Reliable Evaluation framework for Generative event argument extraction (REGen), a framework that better aligns with human judgment. Across six datasets, REGen improves performance by an average of 23.93 F1 points over EM. Human validation further confirms REGen's effectiveness, achieving 87.67% alignment with human assessments of argument correctness.
Code (0)
등록된 구현이 없습니다.
Tasks
Event Argument ExtractionvalidSimilar Papers 제목 키워드 기반
Finding Answers in Thought Matters: Revisiting Evaluation on Large Language Models with Reasoning
Evaluating generative models, such as large language models (LLMs), commonly involves question-answering tasks where the final answer is selected based on probability of answer choices. On the other hand, for models requ…
Question AnsweringHeat Transfer Prediction for Methane in Regenerative Cooling Channels with Neural Networks
Methane is considered being a good choice as a propellant for future reusable launch systems. However, the heat transfer prediction for supercritical methane flowing in cooling channels of a regeneratively cooled combust…
Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models
Diffusion models have revitalized the image generation domain, playing crucial roles in both academic research and artistic expression. With the emergence of new diffusion models, assessing the performance of text-to-ima…
Image GenerationTransduction of an immortalized olfactory ensheathing glia cell line with the green fluorescent protein (GFP) gene: evaluation of its neuroregenerative capacity as a proof of concept
Olfactory ensheathing glia (OEG) cells are known to foster axonal regeneration of central nervous system (CNS) neurons. Several lines of reversibly immortalized human OEG (ihOEG) have been previously established that ena…
Enabling Adoption of Regenerative Agriculture through Soil Carbon Copilots
Mitigating climate change requires transforming agriculture to minimize environ mental impact and build climate resilience. Regenerative agricultural practices enhance soil organic carbon (SOC) levels, thus improving soi…
Management