Reversed in Time: A Novel Temporal-Emphasized Benchmark for Cross-Modal Video-Text Retrieval
Cross-modal (e.g. image-text, video-text) retrieval is an important task in information retrieval and multimodal vision-language understanding field. Temporal understanding makes video-text retrieval more challenging than image-text retrieval. However, we find that the widely used video-text benchmarks have shortcomings in comprehensively assessing abilities of models, especially in temporal understanding, causing large-scale image-text pre-trained models can already achieve comparable zero-shot performance with video-text pre-trained models. In this paper, we introduce RTime, a novel temporal-emphasized video-text retrieval dataset. We first obtain videos of actions or events with significant temporality, and then reverse these videos to create harder negative samples. We then recruit annotators to judge the significance and reversibility of candidate videos, and write captions for qualified videos. We further adopt GPT-4 to extend more captions based on human-written captions. Our RTime dataset currently consists of 21k videos with 10 captions per video, totalling about 122 hours. Based on RTime, we propose three retrieval benchmark tasks: RTime-Origin, RTime-Hard, and RTime-Binary. We further enhance the use of harder-negatives in model training, and benchmark a variety of video-text models on RTime. Extensive experiment analysis proves that RTime indeed poses new and higher challenges to video-text retrieval. We release our RTime dataset\footnote{\url{https://github.com/qyr0403/Reversed-in-Time}} to further advance video-text retrieval and multimodal understanding research.
Code (1)
Tasks
Image-text RetrievalInformation RetrievalRetrievalText RetrievalVideo-Text RetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
What Does a Temporal Benchmark Score Measure? Decomposing Channel Use in Video VLM Evaluation
A score on a temporal video question answering benchmark is meant to measure that a model has temporal understanding, but it conflates two questions. 1. The task question: is the question even temporal, does it need seve…
Video Question AnsweringTemporal Reversed Training for Spiking Neural Networks with Generalized Spatio-Temporal Representation
Spiking neural networks (SNNs) have received widespread attention as an ultra-low power computing paradigm. Recent studies have focused on improving the feature extraction capability of SNNs, but they suffer from ineffic…
3D Point Cloud ClassificationAction RecognitionPoint Cloud ClassificationTime Reversal Symmetry for Efficient Robotic Manipulations in Deep Reinforcement Learning
Symmetry is pervasive in robotics and has been widely exploited to improve sample efficiency in deep reinforcement learning (DRL). However, existing approaches primarily focus on spatial symmetries, such as reflection, r…
Deep Reinforcement Learningreinforcement-learningReinforcement LearningChronological Blindness: Benchmarking Temporal Reasoning in Vision-Language Models with CHRONOSIGHT
Human perception of visual scenes is inherently temporal. We instinctively recognise whether a fruit is ripening or rotting, whether construction is progressing or being demolished, and approximately how much time separa…
Instruction FollowingSee the past: Time-Reversed Scene Reconstruction from Thermal Traces Using Visual Language Models
Recovering the past from present observations is an intriguing challenge with potential applications in forensics and scene analysis. Thermal imaging, operating in the infrared range, provides access to otherwise invisib…
Image Reconstruction