DocLens: Multi-aspect Fine-grained Evaluation for Medical Text Generation
Medical text generation aims to assist with administrative work and highlight salient information to support decision-making. To reflect the specific requirements of medical text, in this paper, we propose a set of metrics to evaluate the completeness, conciseness, and attribution of the generated text at a fine-grained level. The metrics can be computed by various types of evaluators including instruction-following (both proprietary and open-source) and supervised entailment models. We demonstrate the effectiveness of the resulting framework, DocLens, with three evaluators on three tasks: clinical note generation, radiology report summarization, and patient question summarization. A comprehensive human study shows that DocLens exhibits substantially higher agreement with the judgments of medical experts than existing metrics. The results also highlight the need to improve open-source evaluators and suggest potential directions.
Code (2)
Tasks
Decision MakingInstruction FollowingText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
DocLens : A Tool-Augmented Multi-Agent Framework for Long Visual Document Understanding
Comprehending long visual documents, where information is distributed across extensive pages of text and visual elements, is a critical but challenging task for modern Vision-Language Models (VLMs). Existing approaches f…
FRAbench and GenEval: Scaling Fine-Grained Aspect Evaluation across Tasks, Modalities
Evaluating the open-ended outputs of large language models (LLMs) has become a bottleneck as model capabilities, task diversity, and modality coverage rapidly expand. Existing "LLM-as-a-Judge" evaluators are typically na…
Image GenerationText GenerationFine-Grained Opinion Summarization with Minimal Supervision
Opinion summarization aims to profile a target by extracting opinions from multiple documents. Most existing work approaches the task in a semi-supervised manner due to the difficulty of obtaining high-quality annotation…
Fine-Grained Opinion AnalysisOpinion SummarizationSentiment AnalysisFETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation
Recently, open-domain text-to-video (T2V) generation models have made remarkable progress. However, the promising results are mainly shown by the qualitative cases of generated videos, while the quantitative evaluation o…
Text-to-Video GenerationVideo GenerationFINEMATCH: Aspect-based Fine-grained Image and Text Mismatch Detection and Correction
Recent progress in large-scale pre-training has led to the development of advanced vision-language models (VLMs) with remarkable proficiency in comprehending and generating multimodal content. Despite the impressive abil…
HallucinationImage GenerationIn-Context LearningText Matching+2