Progress-Aware Video Frame Captioning
While image captioning provides isolated descriptions for individual images, and video captioning offers one single narrative for an entire video clip, our work explores an important middle ground: progress-aware video captioning at the frame level. This novel task aims to generate temporally fine-grained captions that not only accurately describe each frame but also capture the subtle progression of actions throughout a video sequence. Despite the strong capabilities of existing leading vision language models, they often struggle to discern the nuances of frame-wise differences. To address this, we propose ProgressCaptioner, a captioning model designed to capture the fine-grained temporal dynamics within an action sequence. Alongside, we develop the FrameCap dataset to support training and the FrameCapEval benchmark to assess caption quality. The results demonstrate that ProgressCaptioner significantly surpasses leading captioning models, producing precise captions that accurately capture action progression and set a new standard for temporal precision in video captioning. Finally, we showcase practical applications of our approach, specifically in aiding keyframe selection and advancing video understanding, highlighting its broad utility.
Code (0)
등록된 구현이 없습니다.
Tasks
Image CaptioningVideo CaptioningVideo UnderstandingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
AVC-DPO: Aligned Video Captioning via Direct Preference Optimization
Although video multimodal large language models (video MLLMs) have achieved substantial progress in video captioning tasks, it remains challenging to adjust the focal emphasis of video captions according to human prefere…
Video CaptioningVideo Captioning with Boundary-aware Hierarchical Language Decoding and Joint Video Prediction
The explosion of video data on the internet requires effective and efficient technology to generate captions automatically for people who are not able to watch the videos. Despite the great progress of video captioning r…
DecoderLanguage ModelingLanguage ModellingSentence+3Towards Diverse Paragraph Captioning for Untrimmed Videos
Video paragraph captioning aims to describe multiple events in untrimmed videos with descriptive paragraphs. Existing approaches mainly solve the problem in two steps: event detection and then event captioning. Such two-…
DescriptiveDiversityEvent DetectionSali4Vid: Saliency-Aware Video Reweighting and Adaptive Caption Retrieval for Dense Video Captioning
Dense video captioning aims to temporally localize events in video and generate captions for each event. While recent works propose end-to-end models, they suffer from two limitations: (1) applying timestamp supervision …
Dense Video CaptioningProCap: Prominence-guided Object Rectification for Faithful and Comprehensive Video Captioning
Improving video captioning quality typically demands retraining large vision-language models, an expensive and often impractical requirement. Existing training-free alternatives instead ground captions in detected object…
Video Captioning