KnowIT VQA: Answering Knowledge-Based Questions about Videos
We propose a novel video understanding task by fusing knowledge-based and video question answering. First, we introduce KnowIT VQA, a video dataset with 24,282 human-generated question-answer pairs about a popular sitcom. The dataset combines visual, textual and temporal coherence reasoning together with knowledge-based questions, which need of the experience obtained from the viewing of the series to be answered. Second, we propose a video understanding model by combining the visual and textual video content with specific knowledge about the show. Our main findings are: (i) the incorporation of knowledge produces outstanding improvements for VQA in video, and (ii) the performance on KnowIT VQA still lags well behind human accuracy, indicating its usefulness for studying current video modelling limitations.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringVideo Question AnsweringVideo UnderstandingVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
Knowledge-Based Visual Question Answering in Videos
We propose a novel video understanding task by fusing knowledge-based and video question answering. First, we introduce KnowIT VQA, a video dataset with 24,282 human-generated question-answer pairs about a popular sitcom…
Question AnsweringVideo Question AnsweringVideo UnderstandingVisual Question Answering+1KnowIt: Deep Time Series Modeling and Interpretation
KnowIt (Knowledge discovery in time series data) is a flexible framework for building deep time series models and interpreting them. It is implemented as a Python toolkit, with source code and documentation available fro…
Multi-Modal Retrieval Augmentation for Open-Ended and Knowledge-Intensive Video Question Answering
While current video question answering systems perform well on some tasks requiring only direct visual understanding, they struggle with questions demanding knowledge beyond what is immediately observable in the video co…
Multiple-choiceQuestion AnsweringRetrievalRetrieval-augmented Generation+2NEWSKVQA: Knowledge-Aware News Video Question Answering
Answering questions in the context of videos can be helpful in video indexing, video retrieval systems, video summarization, learning management systems and surveillance video analysis. Although there exists a large body…
Common Sense ReasoningManagementMultiple-choiceQuestion Answering+6CRIPP-VQA: Counterfactual Reasoning about Implicit Physical Properties via Video Question Answering
Videos often capture objects, their visible properties, their motion, and the interactions between different objects. Objects also have physical properties such as mass, which the imaging pipeline is unable to directly c…
Add - POAdd - PQcounterfactualCounterfactual Planning+10