Temporal Action Segmentation with High-level Complex Activity Labels
The temporal action segmentation task segments videos temporally and predicts action labels for all frames. Fully supervising such a segmentation model requires dense frame-wise action annotations, which are expensive and tedious to collect. This work is the first to propose a Constituent Action Discovery (CAD) framework that only requires the video-wise high-level complex activity label as supervision for temporal action segmentation. The proposed approach automatically discovers constituent video actions using an activity classification task. Specifically, we define a finite number of latent action prototypes to construct video-level dual representations with which these prototypes are learned collectively through the activity classification training. This setting endows our approach with the capability to discover potentially shared actions across multiple complex activities. Due to the lack of action-level supervision, we adopt the Hungarian matching algorithm to relate latent action prototypes to ground truth semantic classes for evaluation. We show that with the high-level supervision, the Hungarian matching can be extended from the existing video and activity levels to the global level. The global-level matching allows for action sharing across activities, which has never been considered in the literature before. Extensive experiments demonstrate that our discovered actions can help perform temporal action segmentation and activity recognition tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Action RecognitionAction SegmentationActivity RecognitionSegmentationTemporal Action SegmentationVocal Bursts Intensity PredictionSimilar Papers 제목 키워드 기반
Temporal Convolutional Networks: A Unified Approach to Action Segmentation
The dominant paradigm for video-based action segmentation is composed of two steps: first, for each frame, compute low-level features using Dense Trajectories or a Convolutional Neural Network that encode spatiotemporal …
Action SegmentationSegmentationUnsupervised Skeleton-Based Action Segmentation via Hierarchical Spatiotemporal Vector Quantization
We propose a novel hierarchical spatiotemporal vector quantization framework for unsupervised skeleton-based temporal action segmentation. We first introduce a hierarchical approach, which includes two consecutive levels…
Action SegmentationFrom Stochastic Grammar to Bayes Network: Probabilistic Parsing of Complex Activity
We propose a probabilistic method for parsing a temporal sequence such as a complex activity defined as composition of sub-activities/actions. The temporal structure of the high-level activity is represented by a string-…
Stacked Spatio-Temporal Graph Convolutional Networks for Action Segmentation
We propose novel Stacked Spatio-Temporal Graph Convolutional Networks (Stacked-STGCN) for action segmentation, i.e., predicting and localizing a sequence of actions over long videos. We extend the Spatio-Temporal Graph C…
Action RecognitionAction SegmentationDecoderSkeleton Based Action Recognition+1Hand Guided High Resolution Feature Enhancement for Fine-Grained Atomic Action Segmentation within Complex Human Assemblies
Due to the rapid temporal and fine-grained nature of complex human assembly atomic actions, traditional action segmentation approaches requiring the spatial (and often temporal) down sampling of video frames often loose …
Action ClassificationAction RecognitionAction SegmentationClassification+2