SPICE: Semantic Propositional Image Caption Evaluation
There is considerable interest in the task of automatically generating image
captions. However, evaluation is challenging. Existing automatic evaluation
metrics are primarily sensitive to n-gram overlap, which is neither necessary
nor sufficient for the task of simulating human judgment. We hypothesize that
semantic propositional content is an important component of human caption
evaluation, and propose a new automated caption evaluation metric defined over
scene graphs coined SPICE. Extensive evaluations across a range of models and
datasets indicate that SPICE captures human judgments over model-generated
captions better than other automatic metrics (e.g., system-level correlation of
0.88 with human judgments on the MS COCO dataset, versus 0.43 for CIDEr and
0.53 for METEOR). Furthermore, SPICE can answer questions such as which
caption-generator best understands colors?' and can caption-generators count?'
Code (11)
Tasks
Image CaptioningSimilar Papers 제목 키워드 기반
An Ensemble Model with Attention Based Mechanism for Image Captioning
Image captioning creates informative text from an input image by creating a relationship between the words and the actual content of an image. Recently, deep learning models that utilize transformers have been the most s…
Ensemble LearningImage CaptioningTowards Unique and Informative Captioning of Images
Despite considerable progress, state of the art image captioning models produce generic captions, leaving out important image details. Furthermore, these systems may even misrepresent the image in order to produce a simp…
DiversityImage CaptioningRe-RankingImproved Image Captioning via Policy Gradient optimization of SPIDEr
Current image captioning methods are usually trained via (penalized) maximum likelihood estimation. However, the log-likelihood score of a caption does not correlate well with human assessments of quality. Standard synta…
Image CaptioningJaSPICE: Automatic Evaluation Metric Using Predicate-Argument Structures for Image Captioning Models
Image captioning studies heavily rely on automatic evaluation metrics such as BLEU and METEOR. However, such n-gram-based metrics have been shown to correlate poorly with human evaluation, leading to the proposal of alte…
Image CaptioningTechnical Report of NICE Challenge at CVPR 2024: Caption Re-ranking Evaluation Using Ensembled CLIP and Consensus Scores
This report presents the ECO (Ensembled Clip score and cOnsensus score) pipeline from team DSBA LAB, which is a new framework used to evaluate and rank captions for a given image. ECO selects the most accurate caption de…
Image CaptioningRe-Ranking