Things not Written in Text: Exploring Spatial Commonsense from Visual Signals
Spatial commonsense, the knowledge about spatial position and relationship between objects (like the relative size of a lion and a girl, and the position of a boy relative to a bicycle when cycling), is an important part of commonsense knowledge. Although pretrained language models (PLMs) succeed in many NLP tasks, they are shown to be ineffective in spatial commonsense reasoning. Starting from the observation that images are more likely to exhibit spatial commonsense than texts, we explore whether models with visual signals learn more spatial commonsense than text-based PLMs. We propose a spatial commonsense benchmark that focuses on the relative scales of objects, and the positional relationship between people and objects under different actions. We probe PLMs and models with visual signals, including vision-language pretrained models and image synthesis models, on this benchmark, and find that image synthesis models are more capable of learning accurate and consistent spatial knowledge than other models. The spatial knowledge from image synthesis models also helps in natural language understanding tasks that require spatial commonsense.
Code (1)
Tasks
Image GenerationNatural Language UnderstandingPositionSimilar Papers 제목 키워드 기반
Things not Written in Text: Exploring Spatial Commonsense from Visual Signals
Spatial commonsense, the knowledge about spatial position and relationship between objects (like the relative size of a lion and a girl, and the position of a boy relative to a bicycle when cycling), is an important part…
Image GenerationNatural Language UnderstandingPositionLearning Common Sense Through Visual Abstraction
Common sense is essential for building intelligent machines. While some commonsense knowledge is explicitly stated in human-generated text and can be learnt by mining the web, much of it is unwritten. It is often unneces…
Common Sense ReasoningMultiview Contextual Commonsense Inference: A New Dataset and Task
Contextual commonsense inference is the task of generating various types of explanations around the events in a dyadic dialogue, including cause, motivation, emotional reaction, and others. Producing a coherent and non-t…
Multiview Contextual Commonsense InferenceSafeText: A Benchmark for Exploring Physical Safety in Language Models
Understanding what constitutes safe text is an important issue in natural language processing and can often prevent the deployment of models deemed harmful and unsafe. One such type of safety that has been scarcely studi…
Text GenerationCOMET-M: Reasoning about Multiple Events in Complex Sentences
Understanding the speaker's intended meaning often involves drawing commonsense inferences to reason about what is not stated explicitly. In multi-event sentences, it requires understanding the relationships between even…
coreference-resolutionCoreference ResolutionSentence