A Reconstruction-Based Framework for Caption Evaluation Beyond Reference Captions
Image captioning is a primary task in vision--language research, yet assessing how faithfully a caption preserves image semantics without relying on reference captions remains unsettled. Prevailing evaluations rely on human-annotated references, whose content reflects annotator intent and captioning proficiency. In this paper, we study a reconstruction-based principle for caption evaluation: a caption is as good as its capacity to enable reconstruction of the original image. However, because captioning inherently compresses visual information, it is impossible to recover all details, and pixel-wise comparison between reconstructed and source images is neither feasible nor meaningful. Through our in-depth analysis of the nature of captions, whose fundamental purpose is to transmit the semantic content of an image, we propose a revised principle: a caption is as good as its capacity to enable a reconstruction that is semantically equivalent to the original. To assess semantic equivalence, we test whether the reconstruction matches the original image across a suite of downstream vision--language tasks, yielding a reference-free, task-conditioned caption score. We characterize component-dependent limitations and introduce the lower-cost Captioning Turing Test Dataset (CTTD) surrogate.
Code (0)
등록된 구현이 없습니다.
Tasks
Image CaptioningSimilar Papers 제목 키워드 기반
RefCaptioner: Multi-Reference Image-Grounded Video Captioning
Existing video captioning models generate natural descriptions of video content but cannot explicitly ground local visual elements to multiple reference images. We introduce multi-reference image-grounded video captionin…
Video ReconstructionVideo CaptioningCLIPScore: A Reference-free Evaluation Metric for Image Captioning
Image captioning has conventionally relied on reference-based automatic evaluations, where machine captions are compared against captions written by humans. This is in contrast to the reference-free manner in which human…
Hallucination Pair-wise Detection (1-ref)Hallucination Pair-wise Detection (4-ref)Human Judgment ClassificationHuman Judgment Correlation+1Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extraction
Given the accelerating progress of vision and language modeling, accurate evaluation of machine-generated image captions remains critical. In order to evaluate captions more closely to human preferences, metrics need to …
Image CaptioningLanguage ModelingLanguage ModellingMSD-Score: Multi-Scale Distributional Scoring for Reference-Free Image Caption Evaluation
Evaluating image captions without references remains challenging because global embedding similarity often misses fine-grained mismatches such as hallucinated objects, missing attributes, or incorrect relations. We propo…
Image-text matchingEvaluating Image Caption via Cycle-consistent Text-to-Image Generation
Evaluating image captions typically relies on reference captions, which are costly to obtain and exhibit significant diversity and subjectivity. While reference-free evaluation metrics have been proposed, most focus on c…
Contrastive LearningDiversityImage CaptioningImage Generation+2