Weakly Supervised Energy-Based Learning for Action Segmentation
This paper is about labeling video frames with action classes under weak supervision in training, where we have access to a temporal ordering of actions, but their start and end frames in training videos are unknown. Following prior work, we use an HMM grounded on a Gated Recurrent Unit (GRU) for frame labeling. Our key contribution is a new constrained discriminative forward loss (CDFL) that we use for training the HMM and GRU under weak supervision. While prior work typically estimates the loss on a single, inferred video segmentation, our CDFL discriminates between the energy of all valid and invalid frame labelings of a training video. A valid frame labeling satisfies the ground-truth temporal ordering of actions, whereas an invalid one violates the ground truth. We specify an efficient recursive algorithm for computing the CDFL in terms of the logadd function of the segmentation energy. Our evaluation on action segmentation and alignment gives superior results to those of the state of the art on the benchmark Breakfast Action, Hollywood Extended, and 50Salads datasets.
Code (1)
Tasks
Action SegmentationSegmentationvalidVideo SegmentationVideo Semantic SegmentationWeakly Supervised Action Segmentation (Transcript)Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
FIFA: Fast Inference Approximation for Action Segmentation
We introduce FIFA, a fast approximate inference method for action segmentation and alignment. Unlike previous approaches, FIFA does not rely on expensive dynamic programming for inference. Instead, it uses an approximate…
Action SegmentationSegmentationWeakly Supervised Action Segmentation (Transcript)DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision
We introduce DiscoBox, a novel framework that jointly learns instance segmentation and semantic correspondence using bounding box supervision. Specifically, we propose a self-ensembling framework where instance segmentat…
Box-supervised Instance SegmentationInstance SegmentationSegmentationSemantic correspondence+1Weakly Supervised Segmentation of Cracks on Solar Cells using Normalized Lp Norm
Photovoltaic is one of the most important renewable energy sources for dealing with world-wide steadily increasing energy consumption. This raises the demand for fast and scalable automatic quality management during prod…
General ClassificationManagementSegmentationWeakly-supervised Learning+1On Evaluating Weakly Supervised Action Segmentation Methods
Action segmentation is the task of temporally segmenting every frame of an untrimmed video. Weakly supervised approaches to action segmentation, especially from transcripts have been of considerable interest to the compu…
Action SegmentationSegmentationFast Weakly Supervised Action Segmentation Using Mutual Consistency
Action segmentation is the task of predicting the actions for each frame of a video. As obtaining the full annotation of videos for action segmentation is expensive, weakly supervised approaches that can learn only from …
Action SegmentationSegmentationWeakly Supervised Action Segmentation (Transcript)