Retrieve, Caption, Generate: Visual Grounding for Enhancing Commonsense in Text Generation Models
We investigate the use of multimodal information contained in images as an effective method for enhancing the commonsense of Transformer models for text generation. We perform experiments using BART and T5 on concept-to-text generation, specifically the task of generative commonsense reasoning, or CommonGen. We call our approach VisCTG: Visually Grounded Concept-to-Text Generation. VisCTG involves captioning images representing appropriate everyday scenarios, and using these captions to enrich and steer the generation process. Comprehensive evaluation and analysis demonstrate that VisCTG noticeably improves model performance while successfully addressing several issues of the baseline generations, including poor commonsense, fluency, and specificity.
Code (0)
등록된 구현이 없습니다.
Tasks
Concept-To-Text GenerationSpecificityText GenerationVisual GroundingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CONCAP: Seeing Beyond English with Concepts Retrieval-Augmented Captioning
Multilingual vision-language models have made significant strides in image captioning, yet they still lag behind their English counterparts due to limited multilingual training data and costly large-scale model parameter…
Image CaptioningInference-Free Multimodal Learned Sparse Retrieval for Production-Scale Visual Document Search
As large-scale visual-document corpora such as arXiv papers and enterprise PDFs continue to grow, visual-document retrieval has gained increasing attention; yet it still lacks a deployable system that lexically indexes v…
DualCap: Enhancing Lightweight Image Captioning via Dual Retrieval with Similar Scenes Visual Prompts
Recent lightweight retrieval-augmented image caption models often utilize retrieved data solely as text prompts, thereby creating a semantic gap by leaving the original visual features unenhanced, particularly for object…
Image-to-Text RetrievalImage CaptioningImage RetrievalHierarchical Multi-Modal Retrieval for Knowledge-Grounded News Image Captioning
Traditional image captioning methods often struggle to generate comprehensive, context-rich descriptions, especially for details not directly observable from visual cues. To overcome this, we propose a novel retrieval-au…
Image CaptioningCycle-Consistency Learning for Captioning and Grounding
We present that visual grounding and image captioning, which perform as two mutually inverse processes, can be bridged together for collaborative training by careful designs. By consolidating this idea, we introduce CyCo…
Image CaptioningVisual Grounding