Weakly Supervised Human-Object Interaction Detection in Video via Contrastive Spatiotemporal Regions
We introduce the task of weakly supervised learning for detecting human and object interactions in videos. Our task poses unique challenges as a system does not know what types of human-object interactions are present in a video or the actual spatiotemporal location of the human and the object. To address these challenges, we introduce a contrastive weakly supervised training loss that aims to jointly associate spatiotemporal regions in a video with an action and object vocabulary and encourage temporal continuity of the visual appearance of moving objects as a form of self-supervision. To train our model, we introduce a dataset comprising over 6.5k videos with human-object interaction annotations that have been semi-automatically curated from sentence captions associated with the videos. We demonstrate improved performance over weakly supervised baselines adapted to our task on our video dataset.
Code (1)
Tasks
Human-Object Interaction DetectionObjectSentenceWeakly-supervised LearningSimilar Papers 제목 키워드 기반
Weakly-Supervised HOI Detection from Interaction Labels Only and Language/Vision-Language Priors
Human-object interaction (HOI) detection aims to extract interacting human-object pairs and their interaction categories from a given natural image. Even though the labeling effort required for building HOI detection dat…
Human-Object Interaction DetectionLanguage ModelingLanguage ModellingLarge Language Model+1FreeA: Human-object Interaction Detection using Free Annotation Labels
Recent human-object interaction (HOI) detection methods depend on extensively annotated image datasets, which require a significant amount of manpower. In this paper, we propose a novel self-adaptive, language-driven HOI…
Human-Object Interaction DetectionObjectDetecting Human-Object Interaction with Mixed Supervision
Human object interaction (HOI) detection is an important task in image understanding and reasoning. It is in a form of HOI triplet <human; verb; object>, requiring bounding boxes for human and object, and action between …
Human-Object Interaction DetectionObjectTripletWeakly-supervised LearningTell Me What They're Holding: Weakly-supervised Object Detection with Transferable Knowledge from Human-object Interaction
In this work, we introduce a novel weakly supervised object detection (WSOD) paradigm to detect objects belonging to rare classes that have not many examples using transferable knowledge from human-object interactions (H…
Human-Object Interaction DetectionObjectobject-detectionObject Detection+3RegFormer: Transferable Relational Grounding for Efficient Weakly-Supervised Human-Object Interaction Detection
Weakly-supervised Human-Object Interaction (HOI) detection is essential for scalable scene understanding, as it learns interactions from only image-level annotations. Due to the lack of localization signals, prior works …
Human-Object Interaction DetectionScene Understanding