Visual Semantic Role Labeling
In this paper we introduce the problem of Visual Semantic Role Labeling: given an image we want to detect people doing actions and localize the objects of interaction. Classical approaches to action recognition either study the task of action classification at the image or video clip level or at best produce a bounding box around the person doing the action. We believe such an output is inadequate and a complete understanding can only come when we are able to associate objects in the scene to the different semantic roles of the action. To enable progress towards this goal, we annotate a dataset of 16K people instances in 10K images with actions they are doing and associate objects in the scene with different semantic roles for each action. Finally, we provide a set of baseline algorithms for this task and analyze error modes providing directions for future work.
Code (2)
Tasks
16kAction ClassificationAction RecognitionSemantic Role LabelingTemporal Action LocalizationSimilar Papers 제목 키워드 기반
Grounding Semantic Roles in Images
We address the task of visual semantic role labeling (vSRL), the identification of the participants of a situation or event in a visual scene, and their labeling with their semantic relations to the event or situation. W…
Image CaptioningQuestion AnsweringSemantic Role LabelingVisual Semantic Role Labeling for Video Understanding
We propose a new framework for understanding and representing related salient events in a video using visual semantic role labeling. We represent videos as a set of related events, wherein each event consists of a verb a…
Semantic Role LabelingVideo RecognitionVideo UnderstandingA Flexible and Easy-to-use Semantic Role Labeling Framework for Different Languages
This paper presents a flexible and open source framework for deep semantic role labeling. We aim at facilitating easy exploration of model structures for multiple languages with different characteristics. It provides fle…
Feature EngineeringSemantic Role LabelingAn MRC Framework for Semantic Role Labeling
Semantic Role Labeling (SRL) aims at recognizing the predicate-argument structure of a sentence and can be decomposed into two subtasks: predicate disambiguation and argument labeling. Prior work deals with these two tas…
Computational EfficiencyMachine Reading ComprehensionMultiple-choiceReading Comprehension+2Remove Noise and Keep Truth: A Noisy Channel Model for Semantic Role Labeling
Semantic role labeling usually models structures using sequences, trees, or graphs. Past works focused on researching novel modeling methods and neural structures and integrating more features. In this paper, we re-exami…
Semantic Role Labeling