Seeing the World through Text: Evaluating Image Descriptions for Commonsense Reasoning in Machine Reading Comprehension
Despite recent achievements in natural language understanding, reasoning over commonsense knowledge still represents a big challenge to AI systems. As the name suggests, common sense is related to perception and as such, humans derive it from experience rather than from literary education. Recent works in the NLP and the computer vision field have made the effort of making such knowledge explicit using written language and visual inputs, respectively. Our premise is that the latter source fits better with the characteristics of commonsense acquisition. In this work, we explore to what extent the descriptions of real-world scenes are sufficient to learn common sense about different daily situations, drawing upon visual information to answer script knowledge questions.
Code (0)
등록된 구현이 없습니다.
Tasks
Common Sense ReasoningMachine Reading ComprehensionNatural Language UnderstandingReading ComprehensionSimilar Papers 제목 키워드 기반
The All-Seeing Project V2: Towards General Relation Comprehension of the Open World
We present the All-Seeing Project V2: a new model and dataset designed for understanding object relations in images. Specifically, we propose the All-Seeing Model V2 (ASMv2) that integrates the formulation of text genera…
AllHallucinationObject LocalizationRelation+2Seeing and Seeing Through the Glass: Real and Synthetic Data for Multi-Layer Depth Estimation
Transparent objects are common in daily life, and understanding their multi-layer depth information -- perceiving both the transparent surface and the objects behind it -- is crucial for real-world applications that inte…
Depth EstimationTransparent objectsSeeing The Words: Evaluating AI-generated Biblical Art
The past years witnessed a significant amount of Artificial Intelligence (AI) tools that can generate images from texts. This triggers the discussion of whether AI can generate accurate images using text from the Bible w…
Visual Room 2.0: Seeing is Not Understanding for MLLMs
Can multi-modal large language models (MLLMs) truly understand what they can see? Extending Searle's Chinese Room into the multi-modal domain, this paper proposes the Visual Room argument: MLLMs may describe every visual…
Scene UnderstandingThe All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World
We present the All-Seeing (AS) project: a large-scale data and model for recognizing and understanding everything in the open world. Using a scalable data engine that incorporates human feedback and efficient models in t…
AllQuestion AnsweringRetrievalText Retrieval