HOI-aware Adaptive Network for Weakly-supervised Action Segmentation
In this paper, we propose an HOI-aware adaptive network named AdaAct for weakly-supervised action segmentation. Most existing methods learn a fixed network to predict the action of each frame with the neighboring frames. However, this would result in ambiguity when estimating similar actions, such as pouring juice and pouring coffee. To address this, we aim to exploit temporally global but spatially local human-object interactions (HOI) as video-level prior knowledge for action segmentation. The long-term HOI sequence provides crucial contextual information to distinguish ambiguous actions, where our network dynamically adapts to the given HOI sequence at test time. More specifically, we first design a video HOI encoder that extracts, selects, and integrates the most representative HOI throughout the video. Then, we propose a two-branch HyperNetwork to learn an adaptive temporal encoder, which automatically adjusts the parameters based on the HOI information of various videos on the fly. Extensive experiments on two widely-used datasets including Breakfast and 50Salads demonstrate the effectiveness of our method under different evaluation metrics.
Code (0)
등록된 구현이 없습니다.
Tasks
Action SegmentationSimilar Papers 제목 키워드 기반
DAWN: Domain-Adaptive Weakly Supervised Nuclei Segmentation via Cross-Task Interactions
Weakly supervised segmentation methods have gained significant attention due to their ability to reduce the reliance on costly pixel-level annotations during model training. However, the current weakly supervised nuclei …
Domain AdaptationPseudo LabelSegmentationWeakly supervised segmentationBoxTeacher: Exploring High-Quality Pseudo Labels for Weakly Supervised Instance Segmentation
Labeling objects with pixel-wise segmentation requires a huge amount of human labor compared to bounding boxes. Most existing methods for weakly supervised instance segmentation focus on designing heuristic losses with p…
Box-supervised Instance SegmentationInstance SegmentationSegmentationSemantic Segmentation+2Multi-Miner: Object-Adaptive Region Mining for Weakly-Supervised Semantic Segmentation
Object region mining is a critical step for weakly-supervised semantic segmentation. Most recent methods mine the object regions by expanding the seed regions localized by class activation maps. They generally do not con…
ObjectSegmentationSemantic SegmentationWeakly supervised Semantic Segmentation+1AFANet: Adaptive Frequency-Aware Network for Weakly-Supervised Few-Shot Semantic Segmentation
Few-shot learning aims to recognize novel concepts by leveraging prior knowledge learned from a few samples. However, for visually intensive tasks such as few-shot semantic segmentation, pixel-level annotations are time-…
Few-Shot LearningFew-Shot Semantic SegmentationNovel ConceptsSemantic SegmentationAdaptive Binarization for Weakly Supervised Affordance Segmentation
The concept of affordance is important to understand the relevance of object parts for a certain functional interaction. Affordance types generalize across object categories and are not mutually exclusive. This makes the…
BinarizationObjectSegmentation