MHMS: Multimodal Hierarchical Multimedia Summarization
Multimedia summarization with multimodal output can play an essential role in real-world applications, i.e., automatically generating cover images and titles for news articles or providing introductions to online videos. In this work, we propose a multimodal hierarchical multimedia summarization (MHMS) framework by interacting visual and language domains to generate both video and textual summaries. Our MHMS method contains video and textual segmentation and summarization module, respectively. It formulates a cross-domain alignment objective with optimal transport distance which leverages cross-domain interaction to generate the representative keyframe and textual summary. We evaluated MHMS on three recent multimodal datasets and demonstrated the effectiveness of our method in producing high-quality multimodal summaries.
Code (0)
등록된 구현이 없습니다.
Tasks
ArticlesSimilar Papers 제목 키워드 기반
MSMO: Multimodal Summarization with Multimodal Output
Multimodal summarization has drawn much attention due to the rapid growth of multimedia data. The output of the current multimodal summarization systems is usually represented in texts. However, we have found through exp…
InformativenessText SummarizationMSCMHMST: A traffic flow prediction model based on Transformer
This study proposes a hybrid model based on Transformers, named MSCMHMST, aimed at addressing key challenges in traffic flow prediction. Traditional single-method approaches show limitations in traffic prediction tasks, …
PredictionTraffic PredictionUniMS: A Unified Framework for Multimodal Summarization with Knowledge Distillation
With the rapid increase of multimedia data, a large body of literature has emerged to work on multimodal summarization, the majority of which target at refining salient information from textual and visual modalities to o…
Abstractive Text SummarizationDecoderImage CaptioningKnowledge Distillation+1Leveraging Entity Information for Cross-Modality Correlation Learning: The Entity-Guided Multimodal Summarization
The rapid increase in multimedia data has spurred advancements in Multimodal Summarization with Multimodal Output (MSMO), which aims to produce a multimodal summary that integrates both text and relevant images. The inhe…
Knowledge DistillationLanguage ModelingLanguage ModellingHierarchical3D Adapters for Long Video-to-text Summarization
In this paper, we focus on video-to-text summarization and investigate how to best utilize multimodal information for summarizing long inputs (e.g., an hour-long TV show) into long outputs (e.g., a multi-sentence summary…
SentenceText Summarization