Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extraction
Given the accelerating progress of vision and language modeling, accurate evaluation of machine-generated image captions remains critical. In order to evaluate captions more closely to human preferences, metrics need to discriminate between captions of varying quality and content. However, conventional metrics fail short of comparing beyond superficial matches of words or embedding similarities; thus, they still need improvement. This paper presents VisCE$^2$, a vision language model-based caption evaluation method. Our method focuses on visual context, which refers to the detailed content of images, including objects, attributes, and relationships. By extracting and organizing them into a structured format, we replace the human-written references with visual contexts and help VLMs better understand the image, enhancing evaluation performance. Through meta-evaluation on multiple datasets, we validated that VisCE$^2$ outperforms the conventional pre-trained metrics in capturing caption quality and demonstrates superior consistency with human judgment.
Code (0)
등록된 구현이 없습니다.
Tasks
Image CaptioningLanguage ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
Boosted Attention: Leveraging Human Attention for Image Captioning
Visual attention has shown usefulness in image captioning, with the goal of enabling a caption model to selectively focus on regions of interest. Existing models typically rely on top-down language information and learn …
Image CaptioningFuseCap: Leveraging Large Language Models for Enriched Fused Image Captions
The advent of vision-language pre-training techniques enhanced substantial progress in the development of models for image captioning. However, these models frequently produce generic captions and may omit semantically i…
AttributeImage CaptioningLanguage ModellingLarge Language Model+3VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluation
Video captions play a crucial role in text-to-video generation tasks, as their quality directly influences the semantic coherence and visual fidelity of the generated videos. Although large vision-language models (VLMs) …
Caption GenerationLanguage ModelingLanguage ModellingLarge Language Model+2VisCon-100K: Leveraging Contextual Web Data for Fine-tuning Vision Language Models
Vision-language models (VLMs) excel in various visual benchmarks but are often constrained by the lack of high-quality visual fine-tuning data. To address this challenge, we introduce VisCon-100K, a novel dataset derived…
Image CaptioningLarge Language ModelMultiple-choiceVisual Question Answering (VQA)Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded
Many vision and language models suffer from poor visual grounding - often falling back on easy-to-learn language priors rather than basing their decisions on visual concepts in the image. In this work, we propose a gener…
Image CaptioningQuestion AnsweringVisual GroundingVisual Question Answering+1