ViTOC: Vision Transformer and Object-aware Captioner
This paper presents ViTOC (Vision Transformer and Object-aware Captioner), a novel vision-language model for image captioning that addresses the challenges of accuracy and diversity in generated descriptions. Unlike conventional approaches, ViTOC employs a dual-path architecture based on Vision Transformer and object detector, effectively fusing global visual features and local object information through learnable vectors. The model introduces an innovative object-aware prompting strategy that significantly enhances its capability in handling long-tail data. Experiments on the standard COCO dataset demonstrate that ViTOC outperforms baseline models across all evaluation metrics. Additionally, we propose a reference-free evaluation method based on CLIP to further validate the model's effectiveness. By utilizing pretrained visual model parameters, ViTOC achieves efficient end-to-end training.
Code (0)
등록된 구현이 없습니다.
Tasks
DiversityImage CaptioningLanguage ModelingLanguage ModellingObjectMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
LocCa: Visual Pretraining with Location-aware Captioners
Image captioning has been shown as an effective pretraining method similar to contrastive pretraining. However, the incorporation of location-aware information into visual pretraining remains an area with limited researc…
DecoderImage CaptioningChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions
Asking insightful questions is crucial for acquiring knowledge and expanding our understanding of the world. However, the importance of questioning has been largely overlooked in AI research, where models have been prima…
Image CaptioningQuestion AnsweringVisual ReasoningMore Grounded Image Captioning by Distilling Image-Text Matching Model
Visual attention not only improves the performance of image captioners, but also serves as a visual interpretation to qualitatively measure the caption rationality and model transparency. Specifically, we expect that a c…
Image CaptioningImage-text matchingKnowledge DistillationPOS+2IG Captioner: Information Gain Captioners are Strong Zero-shot Classifiers
Generative training has been demonstrated to be powerful for building visual-language models. However, on zero-shot discriminative benchmarks, there is still a performance gap between models trained with generative and d…
Caption GenerationImage-text RetrievalLanguage ModellingText Retrieval+2OSIC: A New One-Stage Image Captioner Coined
Mainstream image caption models are usually two-stage captioners, i.e., calculating object features by pre-trained detector, and feeding them into a language model to generate text descriptions. However, such an operatio…
DescriptiveLanguage ModellingObjectText Generation