Exploring High-Order Self-Similarity for Video Understanding
Space-time self-similarity (STSS), which captures visual correspondences across frames, provides an effective way to represent temporal dynamics for video understanding. In this work, we explore higher-order STSS and demonstrate how STSSs at different orders reveal distinct aspects of these dynamics. We then introduce the Multi-Order Self-Similarity (MOSS) module, a lightweight neural module designed to learn and integrate multi-order STSS features. It can be applied to diverse video tasks to enhance motion modeling capabilities while consuming only marginal computational cost and memory usage. Extensive experiments on video action recognition, motion-centric video VQA, and real-world robotic tasks consistently demonstrate substantial improvements, validating the broad applicability of MOSS as a general temporal modeling module. The source code and checkpoints will be publicly available.
Code (0)
등록된 구현이 없습니다.
Tasks
Action RecognitionSimilar Papers 제목 키워드 기반
Tensor train rank minimization with nonlocal self-similarity for tensor completion
The tensor train (TT) rank has received increasing attention in tensor completion due to its ability to capture the global correlation of high-order tensors ($\textrm{order} >3$). For third order visual data, direct TT r…
Exploring Global Diversity and Local Context for Video Summarization
Video summarization aims to automatically generate a diverse and concise summary which is useful in large-scale video processing. Most of the methods tend to adopt self-attention mechanism across video frames, which fail…
DiversityVideo SummarizationExploring Temporal Granularity in Self-Supervised Video Representation Learning
This work presents a self-supervised learning framework named TeG to explore Temporal Granularity in learning video representations. In TeG, we sample a long clip from a video and a short clip that lies inside the long c…
Representation LearningSelf-Supervised LearningPatch Spatio-Temporal Relation Prediction for Video Anomaly Detection
Video Anomaly Detection (VAD), aiming to identify abnormalities within a specific context and timeframe, is crucial for intelligent Video Surveillance Systems. While recent deep learning-based VAD models have shown promi…
Anomaly DetectionMulti-Label LearningPredictionRelation+3Self-supervised Spatiotemporal Representation Learning by Exploiting Video Continuity
Recent self-supervised video representation learning methods have found significant success by exploring essential properties of videos, e.g. speed, temporal order, etc. This work exploits an essential yet under-explored…
Action LocalizationAction RecognitionRepresentation LearningRetrieval+1