Seeing What You're Told: Sentence-Guided Activity Recognition In Video
We present a system that demonstrates how the compositional structure of events, in concert with the compositional structure of language, can interplay with the underlying focusing mechanisms in video action recognition, thereby providing a medium, not only for top-down and bottom-up integration, but also for multi-modal integration between vision and language. We show how the roles played by participants (nouns), their characteristics (adjectives), the actions performed (verbs), the manner of such actions (adverbs), and changing spatial relations between participants (prepositions) in the form of whole sentential descriptions mediated by a grammar, guides the activity-recognition process. Further, the utility and expressiveness of our framework is demonstrated by performing three separate tasks in the domain of multi-activity videos: sentence-guided focus of attention, generation of sentential descriptions of video, and query-based video search, simply by leveraging the framework in different manners.
Code (0)
등록된 구현이 없습니다.
Tasks
Action RecognitionActivity RecognitionSentenceTemporal Action LocalizationSimilar Papers 제목 키워드 기반
Tell Me More: A Dataset of Visual Scene Description Sequences
We present a dataset consisting of what we call image description sequences, which are multi-sentence descriptions of the contents of an image. These descriptions were collected in a pseudo-interactive setting, where the…
Image DescriptionSentenceNarrative Interpolation for Generating and Understanding Stories
We propose a method for controlled narrative/story generation where we are able to guide the model to produce coherent narratives with user-specified target endings by interpolation: for example, we are told that Jim wen…
SentenceStory GenerationLearning to Perform Role-Filler Binding with Schematic Knowledge
Through specific experiences, humans learn relationships underlying the structure of events in the world. Schema theory suggests that we organize this information in mental frameworks called "schemata," which represent o…
Question AnsweringSentenceEffective Explanations Support Planning Under Uncertainty
Explaining how to get from A to B can be challenging. It requires mentally simulating what the listener will do based on what they are told. To capture this process, we propose a computational model that converts utteran…
Prompt-Guided Generation of Structured Chest X-Ray Report Using a Pre-trained LLM
Medical report generation automates radiology descriptions from images, easing the burden on physicians and minimizing errors. However, current methods lack structured outputs and physician interactivity for clear, clini…
AnatomyLanguage ModelingLanguage ModellingLarge Language Model+2