Activity Grammars for Temporal Action Segmentation
Sequence prediction on temporal data requires the ability to understand compositional structures of multi-level semantics beyond individual and contextual properties. The task of temporal action segmentation, which aims at translating an untrimmed activity video into a sequence of action segments, remains challenging for this reason. This paper addresses the problem by introducing an effective activity grammar to guide neural predictions for temporal action segmentation. We propose a novel grammar induction algorithm that extracts a powerful context-free grammar from action sequence data. We also develop an efficient generalized parser that transforms frame-level probability distributions into a reliable sequence of actions according to the induced grammar with recursive rules. Our approach can be combined with any neural network for temporal action segmentation to enhance the sequence prediction and discover its compositional structure. Experimental results demonstrate that our method significantly improves temporal action segmentation in terms of both performance and interpretability on two standard benchmarks, Breakfast and 50 Salads.
Code (1)
Tasks
Action SegmentationSegmentationTemporal Action SegmentationSimilar Papers 제목 키워드 기반
Temporal Action Segmentation with High-level Complex Activity Labels
The temporal action segmentation task segments videos temporally and predicts action labels for all frames. Fully supervising such a segmentation model requires dense frame-wise action annotations, which are expensive an…
Action RecognitionAction SegmentationActivity RecognitionSegmentation+2Long-Tail Temporal Action Segmentation with Group-wise Temporal Logit Adjustment
Procedural activity videos often exhibit a long-tailed action distribution due to varying action frequencies and durations. However, state-of-the-art temporal action segmentation methods overlook the long tail and fail t…
Action SegmentationSegmentationTemporal Action SegmentationParsing Videos of Actions with Segmental Grammars
Real-world videos of human activities exhibit temporal structure at various scales; long videos are typically composed out of multiple action instances, where each instance is itself composed of sub-actions with variable…
Generalized Earley Parser: Bridging Symbolic Grammars and Sequence Data for Future Prediction
Future predictions on sequence data (e.g., videos or audios) require the algorithms to capture non-Markovian and compositional properties of high-level semantics. Context-free grammars are natural choices to capture such…
Activity PredictionFuture predictionFrom Stochastic Grammar to Bayes Network: Probabilistic Parsing of Complex Activity
We propose a probabilistic method for parsing a temporal sequence such as a complex activity defined as composition of sub-activities/actions. The temporal structure of the high-level activity is represented by a string-…