EAMA : Entity-Aware Multimodal Alignment Based Approach for News Image Captioning
News image captioning requires model to generate an informative caption rich in entities, with the news image and the associated news article. Current MLLMs still bear limitations in handling entity information in news image captioning tasks. Besides, generating high-quality news image captions requires a trade-off between sufficiency and conciseness of textual input information. To explore the potential of MLLMs and address problems we discovered, we propose EAMA: an Entity-Aware Multimodal Alignment based approach for News Image Captioning. Our approach first aligns the MLLM with two extra alignment tasks: Entity-Aware Sentence Selection task and Entity Selection task, together with News Image Captioning task. The aligned MLLM will utilize the additional entity-related information extracted by itself to supplement the textual input while generating news image captions. Our approach achieves better results than all previous models on two mainstream news image captioning datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Image CaptioningSentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Knowledge Completes the Vision: A Multimodal Entity-aware Retrieval-Augmented Generation Framework for News Image Captioning
News image captioning aims to produce journalistically informative descriptions by combining visual content with contextual cues from associated articles. Despite recent advances, existing methods struggle with three key…
Image CaptioningMultimodal Matching-aware Co-attention Networks with Mutual Knowledge Distillation for Fake News Detection
Fake news often involves multimedia information such as text and image to mislead readers, proliferating and expanding its influence. Most existing fake news detection methods apply the co-attention mechanism to fuse mul…
Fake News DetectionImage-text matchingKnowledge DistillationText MatchingEntityCLIP: Entity-Centric Image-Text Matching via Multimodal Attentive Contrastive Learning
Recent advancements in image-text matching have been notable, yet prevailing models predominantly cater to broad queries and struggle with accommodating fine-grained query intention. In this paper, we work towards the \t…
Contrastive LearningImage-text matchingText MatchingPrompt Me Up: Unleashing the Power of Alignments for Multimodal Entity and Relation Extraction
How can we better extract entities and relations from text? Using multimodal extraction with images and text obtains more signals for entities and relations, and aligns them through graphs or hierarchical fusion, aiding …
RelationRelation ExtractionStreamAlign: Streaming Text-Aligned Speech Tokenization
Text-aligned speech tokenization methods have emerged to better align speech tokens with LLM token spaces, enabling more effective utilization of pretrained LLMs. However, they rely on offline automatic speech recognitio…
Speech Recognition