CLEVRER: CoLlision Events for Video REpresentation and Reasoning
The ability to reason about temporal and causal events from videos lies at the core of human intelligence. Most video reasoning benchmarks, however, focus on pattern recognition from complex visual and language input, instead of on causal structure. We study the complementary problem, exploring the temporal and causal structures behind videos of objects with simple visual appearance. To this end, we introduce the CoLlision Events for Video REpresentation and Reasoning (CLEVRER), a diagnostic video dataset for systematic evaluation of computational models on a wide range of reasoning tasks. Motivated by the theory of human casual judgment, CLEVRER includes four types of questions: descriptive (e.g., "what color"), explanatory ("what is responsible for"), predictive ("what will happen next"), and counterfactual ("what if"). We evaluate various state-of-the-art models for visual reasoning on our benchmark. While these models thrive on the perception-based task (descriptive), they perform poorly on the causal tasks (explanatory, predictive and counterfactual), suggesting that a principled approach for causal reasoning should incorporate the capability of both perceiving complex visual and language inputs, and understanding the underlying dynamics and causal relations. We also study an oracle model that explicitly combines these components via symbolic representations.
Code (3)
Tasks
counterfactualDescriptiveDiagnosticVisual ReasoningSimilar Papers 제목 키워드 기반
Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning
We study the problem of dynamic visual reasoning on raw videos. This is a challenging problem; currently, state-of-the-art models often require dense supervision on physical object properties and events from simulation, …
counterfactualObjectRetrievalVideo Retrieval+1Grounding Physical Object and Event Concepts Through Dynamic Visual Reasoning
We study the problem of dynamic visual reasoning on raw videos. This is a challenging problem; currently, state-of-the-art models often require dense supervision on physical object properties and events from simulation, …
counterfactualObjectRetrievalVideo Retrieval+1CLEVRER-Humans: Describing Physical and Causal Events the Human Way
Building machines that can reason about physical events and their causal relationships is crucial for flexible interaction with the physical world. However, most existing physical and causal reasoning benchmarks are excl…
Causal JudgmentData AugmentationDiversityQuestion AnsweringThink before You Simulate: Symbolic Reasoning to Orchestrate Neural Computation for Counterfactual Question Answering
Causal and temporal reasoning about video dynamics is a challenging problem. While neuro-symbolic models that combine symbolic reasoning with neural-based perception and prediction have shown promise, they exhibit limita…
counterfactualCounterfactual ReasoningQuestion AnsweringHySTER: A Hybrid Spatio-Temporal Event Reasoner
The task of Video Question Answering (VideoQA) consists in answering natural language questions about a video and serves as a proxy to evaluate the performance of a model in scene sequence understanding. Most methods des…
Inductive logic programmingQuestion AnsweringVideo Question Answering