Improving Image Captioning Evaluation by Considering Inter References Variance
Evaluating image captions is very challenging partially due to the fact that there are multiple correct captions for every single image. Most of the existing one-to-one metrics operate by penalizing mismatches between reference and generative caption without considering the intrinsic variance between ground truth captions. It usually leads to over-penalization and thus a bad correlation to human judgment. Recently, the latest one-to-one metric BERTScore can achieve high human correlation in system-level tasks while some issues can be fixed for better performance. In this paper, we propose a novel metric based on BERTScore that could handle such a challenge and extend BERTScore with a few new features appropriately for image captioning evaluation. The experimental results show that our metric achieves state-of-the-art human judgment correlation.
Code (0)
등록된 구현이 없습니다.
Tasks
Human Judgment CorrelationImage CaptioningSimilar Papers 제목 키워드 기반
Multi-Reference Training with Pseudo-References for Neural Translation and Text Generation
Neural text generation, including neural machine translation, image captioning, and summarization, has been quite successful recently. However, during training time, typically only one reference is considered for each ex…
Image CaptioningMachine TranslationText GenerationTranslationTowards Annotation-Free Evaluation of Cross-Lingual Image Captioning
Cross-lingual image captioning, with its ability to caption an unlabeled image in a target language other than English, is an emerging topic in the multimedia field. In order to save the precious human resource from re-w…
Image CaptioningMachine TranslationTranslationREO-Relevance, Extraness, Omission: A Fine-grained Evaluation for Image Captioning
Popular metrics used for evaluating image captioning systems, such as BLEU and CIDEr, provide a single score to gauge the system's overall effectiveness. This score is often not informative enough to indicate what specif…
Image CaptioningLOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences
Large Vision-Language Models (LVLMs) have transformed image captioning, shifting from concise captions to detailed descriptions. We introduce LOTUS, a leaderboard for evaluating detailed captions, addressing three main g…
Image CaptioningEMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matching
Current metrics for video captioning are mostly based on the text-level comparison between reference and candidate captions. However, they have some insuperable drawbacks, e.g., they cannot handle videos without referenc…
Language ModellingVideo Captioning