ViLBERTScore: Evaluating Image Caption Using Vision-and-Language BERT
In this paper, we propose an evaluation metric for image captioning systems using both image and text information. Unlike the previous methods that rely on textual representations in evaluating the caption, our approach uses visiolinguistic representations. The proposed method generates image-conditioned embeddings for each token using ViLBERT from both generated and reference texts. Then, these contextual embeddings from each of the two sentence-pair are compared to compute the similarity score. Experimental results on three benchmark datasets show that our method correlates significantly better with human judgments than all existing metrics.
Code (1)
Tasks
Image CaptioningSentenceSimilar Papers 제목 키워드 기반
Prompt-Based Caption Generation for Single-Tooth Dental Images Using Vision-Language Models
Digital dentistry has made significant advances with the advent of deep learning. However, the majority of these deep learning-based dental image analysis models focus on very specific tasks such as tooth segmentation, t…
Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning
Vision-language models (VLMs) mainly rely on contrastive training to learn general-purpose representations of images and captions. We focus on the situation when one image is associated with several captions, each captio…
Contrastive LearningRepresentation Learning"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models
Vision-Language Models (VLMs) are increasingly used by blind and low-vision (BLV) people to identify and understand products in their everyday lives, such as food, personal care items, and household goods. Despite their …
A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions
Curation methods for massive vision-language datasets trade off between dataset size and quality. However, even the highest quality of available curated captions are far too short to capture the rich visual detail in an …
Image CaptioningUIT-OpenViIC: A Novel Benchmark for Evaluating Image Captioning in Vietnamese
Image Captioning is one of the vision-language tasks that still interest the research community worldwide in the 2020s. MS-COCO Caption benchmark is commonly used to evaluate the performance of advanced captioning models…
Image CaptioningVietnamese Image CaptioningVietnamese Multimodal Learning