Can You Spot the Semantic Predicate in this Video?
We propose a method to improve human activity recognition in video by leveraging semantic information about the target activities from an expert-defined linguistic resource, VerbNet. Our hypothesis is that activities that share similar event semantics, as defined by the semantic predicates of VerbNet, will be more likely to share some visual components. We use a deep convolutional neural network approach as a baseline and incorporate linguistic information from VerbNet through multi-task learning. We present results of experiments showing the added information has negligible impact on recognition performance. We discuss how this may be because the lexical semantic information defined by VerbNet is generally not visually salient given the video processing approach used here, and how we may handle this in future approaches.
Code (0)
등록된 구현이 없습니다.
Tasks
Activity RecognitionHuman Activity RecognitionMulti-Task LearningSimilar Papers 제목 키워드 기반
SPOT! Revisiting Video-Language Models for Event Understanding
Understanding videos is an important research topic for multimodal learning. Leveraging large-scale datasets of web-crawled video-text pairs as weak supervision has become a pre-training paradigm for learning joint repre…
AttributeVideo UnderstandingSMART: MLLM-guided Temporal Alignment for Unifying Sign Language Recognition and Spotting
Continuous sign language recognition (CSLR) aims to recognize gloss sequences from unsegmented sign videos under weak sequence-level supervision. However, existing methods rely on sentence-level gloss annotations, provid…
Sign Language RecognitionRepresentation LearningA Context-Aware Loss Function for Action Spotting in Soccer Videos
In video understanding, action spotting consists in temporally localizing human-induced events annotated with single timestamps. In this paper, we propose a novel loss function that specifically considers the temporal co…
Action SpottingVideo UnderstandingGrounded Human-Object Interaction Hotspots from Video (Extended Abstract)
Learning how to interact with objects is an important step towards embodied visual intelligence, but existing techniques suffer from heavy supervision or sensing requirements. We propose an approach to learn human-object…
Human-Object Interaction DetectionObjectSemantic SegmentationCross-Modal Graph with Meta Concepts for Video Captioning
Video captioning targets interpreting the complex visual contents as text descriptions, which requires the model to fully understand video scenes including objects and their interactions. Prevailing methods adopt off-the…
object-detectionObject DetectionVideo Captioning