UBiSS: A Unified Framework for Bimodal Semantic Summarization of Videos
With the surge in the amount of video data, video summarization techniques, including visual-modal(VM) and textual-modal(TM) summarization, are attracting more and more attention. However, unimodal summarization inevitably loses the rich semantics of the video. In this paper, we focus on a more comprehensive video summarization task named Bimodal Semantic Summarization of Videos (BiSSV). Specifically, we first construct a large-scale dataset, BIDS, in (video, VM-Summary, TM-Summary) triplet format. Unlike traditional processing methods, our construction procedure contains a VM-Summary extraction algorithm aiming to preserve the most salient content within long videos. Based on BIDS, we propose a Unified framework UBiSS for the BiSSV task, which models the saliency information in the video and generates a TM-summary and VM-summary simultaneously. We further optimize our model with a list-wise ranking-based objective to improve its capacity to capture highlights. Lastly, we propose a metric, $NDCG_{MS}$, to provide a joint evaluation of the bimodal summary. Experiments show that our unified framework achieves better performance than multi-stage summarization pipelines. Code and data are available at https://github.com/MeiYutingg/UBiSS.
Code (1)
Tasks
TripletVideo SummarizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
BPFNet: A Unified Framework for Bimodal Palmprint Alignment and Fusion
Bimodal palmprint recognition leverages palmprint and palm vein images simultaneously,which achieves high accuracy by multi-model information fusion and has strong anti-falsification property. In the recognition pipeline…
Keypoint DetectionTranslationA Second Look on BASS -- Boosting Abstractive Summarization with Unified Semantic Graphs -- A Replication Study
We present a detailed replication study of the BASS framework, an abstractive summarization system based on the notion of Unified Semantic Graphs. Our investigation includes challenges in replicating key components and a…
Abstractive Text SummarizationBASS: Boosting Abstractive Summarization with Unified Semantic Graph
Abstractive summarization for long-document or multi-document remains challenging for the Seq2Seq architecture, as Seq2Seq is not good at analyzing long-distance relations in text. In this paper, we present BASS, a novel…
Abstractive Text SummarizationDecoderDocument SummarizationImplicit Relations+1Retrieval Augmented Code Generation and Summarization
Software developers write a lot of source code and documentation during software development. Intrinsically, developers often recall parts of source code or code summaries that they had written in the past while implemen…
Code GenerationCode SummarizationRetrievalBiOcularGAN: Bimodal Synthesis and Annotation of Ocular Images
Current state-of-the-art segmentation techniques for ocular images are critically dependent on large-scale annotated datasets, which are labor-intensive to gather and often raise privacy concerns. In this paper, we prese…
Image GenerationSegmentation