Language-Guided Self-Supervised Video Summarization Using Text Semantic Matching Considering the Diversity of the Video
Current video summarization methods rely heavily on supervised computer vision techniques, which demands time-consuming and subjective manual annotations. To overcome these limitations, we investigated self-supervised video summarization. Inspired by the success of Large Language Models (LLMs), we explored the feasibility in transforming the video summarization task into a Natural Language Processing (NLP) task. By leveraging the advantages of LLMs in context understanding, we aim to enhance the effectiveness of self-supervised video summarization. Our method begins by generating captions for individual video frames, which are then synthesized into text summaries by LLMs. Subsequently, we measure semantic distance between the captions and the text summary. Notably, we propose a novel loss function to optimize our model according to the diversity of the video. Finally, the summarized video can be generated by selecting the frames with captions similar to the text summary. Our method achieves state-of-the-art performance on the SumMe dataset in rank correlation coefficients. In addition, our method has a novel feature of being able to achieve personalized summarization.
Code (0)
등록된 구현이 없습니다.
Tasks
DiversitySupervised Video SummarizationVideo SummarizationSimilar Papers 제목 키워드 기반
Language-guided Recursive Spatiotemporal Graph Modeling for Video Summarization
Video summarization aims to select keyframes that are visually diverse and can represent the whole story of a given video. Previous approaches have focused on global interlinkability between frames in a video by temporal…
Video SummarizationTRIMMER: A New Paradigm for Video Summarization through Self-Supervised Reinforcement Learning
The rapid growth of video content across domains such as surveillance, education, and social media has made efficient content understanding increasingly critical. Video summarization addresses this challenge by generatin…
Self-Supervised LearningComputational EfficiencyReinforcement LearningVideo SummarizationCLIP-It! Language-Guided Video Summarization
A generic video summary is an abridged version of a video that conveys the whole story and features the most important scenes. Yet the importance of scenes in a video is often subjective, and users should have the option…
Query-focused SummarizationQuery focused video summarizationSupervised Video SummarizationVideo SummarizationSelf-Attention Recurrent Summarization Network with Reinforcement Learning for Video Summarization Task
With the exponential growth of video data, video summarization techniques are urgently needed for reducing people’s efforts in the videos' content exploration by generating succinct but informative summaries from origina…
reinforcement-learningReinforcement LearningSupervised Video SummarizationUnsupervised Video Summarization+1Prompts to Summaries: Zero-Shot Language-Guided Video Summarization
The explosive growth of video data intensified the need for flexible user-controllable summarization tools that can operate without domain-specific training data. Existing methods either rely on datasets, limiting genera…
GPUQuery focused video summarizationVideo Summarization