Video Action Differencing
How do two individuals differ when performing the same action? In this work, we introduce Video Action Differencing (VidDiff), the novel task of identifying subtle differences between videos of the same action, which has many applications, such as coaching and skill learning. To enable development on this new task, we first create VidDiffBench, a benchmark dataset containing 549 video pairs, with human annotations of 4,469 fine-grained action differences and 2,075 localization timestamps indicating where these differences occur. Our experiments demonstrate that VidDiffBench poses a significant challenge for state-of-the-art large multimodal models (LMMs), such as GPT-4o and Qwen2-VL. By analyzing failure cases of LMMs on VidDiffBench, we highlight two key challenges for this task: localizing relevant sub-actions over two videos and fine-grained frame comparison. To overcome these, we propose the VidDiff method, an agentic workflow that breaks the task into three stages: action difference proposal, keyframe localization, and frame differencing, each stage utilizing specialized foundation models. To encourage future research in this new task, we release the benchmark at https://huggingface.co/datasets/jmhb/VidDiffBench and code at http://jmhb0.github.io/viddiff.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Micro-Expression-Aware Avatar Fingerprinting via Inter-Frame Feature Differencing
Avatar fingerprinting, i.e., verifying who drives a synthetic talking-head video rather than whether it is real, is a critical safeguard for authorized use of face-reenactment technology. Existing methods rely on a fixed…
Target Tracking In Real Time Surveillance Cameras and Videos
Security concerns has been kept on increasing, so it is important for everyone to keep their property safe from thefts and destruction. So the need for surveillance techniques are also increasing. The system has been dev…
Video Background SubtractionAnomaly detection in non-stationary videos using time-recursive differencing network based prediction
Most videos, including those captured through aerial remote sensing, are usually non-stationary in nature having time-varying feature statistics. Although, sophisticated reconstruction and prediction models exist for vid…
Anomaly DetectionOptical Flow EstimationPredictionVideo Anomaly DetectionCapturing Video Frame Rate Variations via Entropic Differencing
High frame rate videos are increasingly getting popular in recent years, driven by the strong requirements of the entertainment and streaming industries to provide high quality of experiences to consumers. To achieve the…
Video Quality AssessmentHierarchical Feature Aggregation Networks for Video Action Recognition
Most action recognition methods base on a) a late aggregation of frame level CNN features using average pooling, max pooling, or RNN, among others, or b) spatio-temporal aggregation via 3D convolutions. The first assume …
Action RecognitionTemporal Action Localization