Detecting Human-Object Interaction with Mixed Supervision
Human object interaction (HOI) detection is an important task in image understanding and reasoning. It is in a form of HOI triplet <human; verb; object>, requiring bounding boxes for human and object, and action between them for the task completion. In other words, this task requires strong supervision for training that is however hard to procure. A natural solution to overcome this is to pursue weakly-supervised learning, where we only know the presence of certain HOI triplets in images but their exact location is unknown. Most weakly-supervised learning methods do not make provision for leveraging data with strong supervision, when they are available; and indeed a na\"ive combination of this two paradigms in HOI detection fails to make contributions to each other. In this regard we propose a mixed-supervised HOI detection pipeline: thanks to a specific design of momentum-independent learning that learns seamlessly across these two types of supervision. Moreover, in light of the annotation insufficiency in mixed supervision, we introduce an HOI element swapping technique to synthesize diverse and hard negatives across images and improve the robustness of the model. Our method is evaluated on the challenging HICO-DET dataset. It performs close to or even better than many fully-supervised methods by using a mixed amount of strong and weak annotations; furthermore, it outperforms representative state of the art weakly and fully-supervised methods under the same supervision.
Code (0)
등록된 구현이 없습니다.
Tasks
Human-Object Interaction DetectionObjectTripletWeakly-supervised LearningSimilar Papers 제목 키워드 기반
Learning Transferable Human-Object Interaction Detector With Natural Language Supervision
It is difficult to construct a data collection including all possible combinations of human actions and interacting objects due to the combinatorial nature of human-object interactions (HOI). In this work, we aim to …
Human-Object Interaction DetectionWeakly Supervised Human-Object Interaction Detection in Video via Contrastive Spatiotemporal Regions
We introduce the task of weakly supervised learning for detecting human and object interactions in videos. Our task poses unique challenges as a system does not know what types of human-object interactions are present in…
Human-Object Interaction DetectionObjectSentenceWeakly-supervised LearningExploring Pose-Aware Human-Object Interaction via Hybrid Learning
Human-Object Interaction (HOI) detection plays a crucial role in visual scene comprehension. In recent advancements two-stage detectors have taken a prominent position. However they are encumbered by two primary chal…
Human-Object Interaction DetectionObjectHuman-Object Interaction Detection via Weak Supervision
The goal of this paper is Human-object Interaction (HO-I) detection. HO-I detection aims to find interacting human-objects regions and classify their interaction from an image. Researchers obtain significant improvement …
Human-Object Interaction DetectionObjectGID-Net: Detecting Human-Object Interaction with Global and Instance Dependency
Since detecting and recognizing individual human or object are not adequate to understand the visual world, learning how humans interact with surrounding objects becomes a core technology. However, convolution operations…
Human-Object Interaction DetectionObject