In Defense of Clip-based Video Relation Detection
Video Visual Relation Detection (VidVRD) aims to detect visual relationship triplets in videos using spatial bounding boxes and temporal boundaries. Existing VidVRD methods can be broadly categorized into bottom-up and top-down paradigms, depending on their approach to classifying relations. Bottom-up methods follow a clip-based approach where they classify relations of short clip tubelet pairs and then merge them into long video relations. On the other hand, top-down methods directly classify long video tubelet pairs. While recent video-based methods utilizing video tubelets have shown promising results, we argue that the effective modeling of spatial and temporal context plays a more significant role than the choice between clip tubelets and video tubelets. This motivates us to revisit the clip-based paradigm and explore the key success factors in VidVRD. In this paper, we propose a Hierarchical Context Model (HCM) that enriches the object-based spatial context and relation-based temporal context based on clips. We demonstrate that using clip tubelets can achieve superior performance compared to most video-based methods. Additionally, using clip tubelets offers more flexibility in model designs and helps alleviate the limitations associated with video tubelets, such as the challenging long-term object tracking problem and the loss of temporal information in long-term tubelet feature compression. Extensive experiments conducted on two challenging VidVRD benchmarks validate that our HCM achieves a new state-of-the-art performance, highlighting the effectiveness of incorporating advanced spatial and temporal context modeling within the clip-based paradigm.
Code (0)
등록된 구현이 없습니다.
Tasks
Feature CompressionObject TrackingRelationVideo Visual Relation DetectionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Long Short-Term Relation Networks for Video Action Detection
It has been well recognized that modeling human-object or object-object relations would be helpful for detection task. Nevertheless, the problem is not trivial especially when exploring the interactions between human act…
Action DetectionObjectRegion ProposalRelation+1Correlation-Guided Query-Dependency Calibration for Video Temporal Grounding
Temporal Grounding is to identify specific moments or highlights from a video corresponding to textual descriptions. Typical approaches in temporal grounding treat all video clips equally during the encoding process rega…
Highlight DetectionMoment RetrievalNatural Language Moment RetrievalRepresentation Learning+1End-to-end Open-vocabulary Video Visual Relationship Detection using Multi-modal Prompting
Open-vocabulary video visual relationship detection aims to expand video visual relationship detection beyond annotated categories by detecting unseen relationships between both seen and unseen objects in videos. Existin…
DecoderObjectobject-detectionObject Detection+5Jailbreaking Multimodal Large Language Models using Multi-Clip Video
As multimodal large language models (MLLMs) have advanced to process video inputs, concerns have emerged about their potential for malicious misuse. Prior jailbreak studies have shown that safety alignment in MLLMs can b…
Multiple Instance-Based Video Anomaly Detection using Deep Temporal Encoding-Decoding
In this paper, we propose a weakly supervised deep temporal encoding-decoding solution for anomaly detection in surveillance videos using multiple instance learning. The proposed approach uses both abnormal and normal vi…
Anomaly DetectionAnomaly Detection In Surveillance VideosMultiple Instance LearningVideo Anomaly Detection