Abstractive Text-Image Summarization Using Multi-Modal Attentional Hierarchical RNN
Rapid growth of multi-modal documents on the Internet makes multi-modal summarization research necessary. Most previous research summarizes texts or images separately. Recent neural summarization research shows the strength of the Encoder-Decoder model in text summarization. This paper proposes an abstractive text-image summarization model using the attentional hierarchical Encoder-Decoder model to summarize a text document and its accompanying images simultaneously, and then to align the sentences and images in summaries. A multi-modal attentional mechanism is proposed to attend original sentences, images, and captions when decoding. The DailyMail dataset is extended by collecting images and captions from the Web. Experiments show our model outperforms the neural abstractive and extractive text summarization methods that do not consider images. In addition, our model can generate informative summaries of images.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderExtractive Text SummarizationText SummarizationSimilar Papers 제목 키워드 기반
UniMS: A Unified Framework for Multimodal Summarization with Knowledge Distillation
With the rapid increase of multimedia data, a large body of literature has emerged to work on multimodal summarization, the majority of which target at refining salient information from textual and visual modalities to o…
Abstractive Text SummarizationDecoderImage CaptioningKnowledge Distillation+1MAST: Multimodal Abstractive Summarization with Trimodal Hierarchical Attention
This paper presents MAST, a new model for Multimodal Abstractive Text Summarization that utilizes information from all three modalities -- text, audio and video -- in a multimodal video. Prior work on multimodal abstract…
Abstractive Text SummarizationMultimodal Abstractive Text SummarizationText SummarizationMultimodal Abstractive Summarization for How2 Videos
In this paper, we study abstractive summarization for open-domain videos. Unlike the traditional text news summarization, the goal is less to "compress" text information but rather to provide a fluent textual summary of …
Abstractive Text SummarizationNews SummarizationText SummarizationSummary-Oriented Vision Modeling for Multimodal Abstractive Summarization
Multimodal abstractive summarization (MAS) aims to produce a concise summary given the multimodal data (text and vision). Existing studies mainly focus on how to effectively use the visual features from the perspective o…
Abstractive Text SummarizationSee, Hear, Read: Leveraging Multimodality with Guided Attention for Abstractive Text Summarization
In recent years, abstractive text summarization with multimodal inputs has started drawing attention due to its ability to accumulate information from different source modalities and generate a fluent textual summary. Ho…
Abstractive Text SummarizationDecoderLanguage ModellingText Summarization