Attention-Based Context Aware Reasoning for Situation Recognition
Situation Recognition (SR) is a fine-grained action recognition task where the model is expected to not only predict the salient action of the image, but also predict values of all associated semantic roles of the action. Predicting semantic roles is very challenging: a vast variety of possibilities can be the match for a semantic role. Existing work has focused on dependency modelling architectures to solve this issue. Inspired by the success achieved by query-based visual reasoning (e.g., Visual Question Answering), we propose to address semantic role prediction as a query-based visual reasoning problem. However, existing query-based reasoning methods have not considered handling of inter-dependent queries which is a unique requirement of semantic role prediction in SR. Therefore, to the best of our knowledge, we propose the first set of methods to address inter-dependent queries in query-based visual reasoning. Extensive experiments demonstrate the effectiveness of our proposed method which achieves outstanding performance on Situation Recognition task. Furthermore, leveraging query inter-dependency, our methods improve upon a state-of-the-art method that answers queries separately. Our code: https://github.com/thilinicooray/context-aware-reasoning-for-sr
Code (1)
Tasks
Action RecognitionFine-grained Action RecognitionGrounded Situation RecognitionQuestion AnsweringSituation RecognitionVisual Question AnsweringVisual Question Answering (VQA)Visual ReasoningSimilar Papers 제목 키워드 기반
Mixture-Kernel Graph Attention Network for Situation Recognition
Understanding images beyond salient actions involves reasoning about scene context, objects, and the roles they play in the captured event. Situation recognition has recently been introduced as the task of jointly reason…
Graph AttentionGraph Neural NetworkGrounded Situation RecognitionSituation RecognitionThe Reasoning Trap -- Logical Reasoning as a Mechanistic Pathway to Situational Awareness
Situational awareness, the capacity of an AI system to recognize its own nature, understand its training and deployment context, and reason strategically about its circumstances, is widely considered among the most dange…
Logical ReasoningTaken out of context: On measuring situational awareness in LLMs
We aim to better understand the emergence of `situational awareness' in large language models (LLMs). A model is situationally aware if it's aware that it's a model and can recognize whether it's currently in testing or …
Data AugmentationIn-Context LearningIntentQA: Context-aware Video Intent Reasoning
In this paper, we propose a novel task IntentQA, a special VideoQA task focusing on video intent reasoning, which has become increasingly important for AI with its advantages in equipping AI agents with the capabilit…
Contrastive LearningVideo Question AnsweringAssessing Drivers' Situation Awareness in Semi-Autonomous Vehicles: ASP based Characterisations of Driving Dynamics for Modelling Scene Interpretation and Projection
Semi-autonomous driving, as it is already available today and will eventually become even more accessible, implies the need for driver and automation system to reliably work together in order to ensure safe driving. A pa…
Autonomous DrivingAutonomous Vehicles