Exploring Self- and Cross-Triplet Correlations for Human-Object Interaction Detection
Human-Object Interaction (HOI) detection plays a vital role in scene understanding, which aims to predict the HOI triplet in the form of <human, object, action>. Existing methods mainly extract multi-modal features (e.g., appearance, object semantics, human pose) and then fuse them together to directly predict HOI triplets. However, most of these methods focus on seeking for self-triplet aggregation, but ignore the potential cross-triplet dependencies, resulting in ambiguity of action prediction. In this work, we propose to explore Self- and Cross-Triplet Correlations (SCTC) for HOI detection. Specifically, we regard each triplet proposal as a graph where Human, Object represent nodes and Action indicates edge, to aggregate self-triplet correlation. Also, we try to explore cross-triplet dependencies by jointly considering instance-level, semantic-level, and layout-level relations. Besides, we leverage the CLIP model to assist our SCTC obtain interaction-aware feature by knowledge distillation, which provides useful action clues for HOI detection. Extensive experiments on HICO-DET and V-COCO datasets verify the effectiveness of our proposed SCTC.
Code (0)
등록된 구현이 없습니다.
Tasks
Human-Object Interaction DetectionKnowledge DistillationObjectScene UnderstandingTripletMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Exploring Correlations of Self-Supervised Tasks for Graphs
Graph self-supervised learning has sparked a research surge in training informative representations without accessing any labeled data. However, our understanding of graph self-supervised learning remains limited, and th…
Multi-Task LearningSelf-Supervised LearningTriplet Attention Transformer for Spatiotemporal Predictive Learning
Spatiotemporal predictive learning offers a self-supervised learning paradigm that enables models to learn both spatial and temporal patterns by predicting future sequences based on historical sequences. Mainstream metho…
Computational EfficiencyPredictionSelf-Supervised LearningTrajectory Prediction+1Deep Comprehensive Correlation Mining for Image Clustering
Recent developed deep unsupervised methods allow us to jointly learn representation and cluster unlabelled data. These deep clustering methods mainly focus on the correlation among samples, e.g., selecting high precision…
ClusteringDeep ClusteringImage ClusteringPseudo Label+1Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection
Human-Object Interaction (HOI) detection is a challenging computer vision task that requires visual models to address the complex interactive relationship between humans and objects and predict HOI triplets. Despite the …
Human-Object Interaction DetectionSentenceTransfer LearningCross-Modal Subspace Learning with Scheduled Adaptive Margin Constraints
Cross-modal embeddings, between textual and visual modalities, aim to organise multimodal instances by their semantic correlations. State-of-the-art approaches use maximum-margin methods, based on the hinge-loss, to enfo…
Incremental LearningTriplet