Exploring The Spatial Reasoning Ability of Neural Models in Human IQ Tests
Although neural models have performed impressively well on various tasks such as image recognition and question answering, their reasoning ability has been measured in only few studies. In this work, we focus on spatial reasoning and explore the spatial understanding of neural models. First, we describe the following two spatial reasoning IQ tests: rotation and shape composition. Using well-defined rules, we constructed datasets that consist of various complexity levels. We designed a variety of experiments in terms of generalization, and evaluated six different baseline models on the newly generated datasets. We provide an analysis of the results and factors that affect the generalization abilities of models. Also, we analyze how neural models solve spatial reasoning tests with visual aids. Our findings would provide valuable insights into understanding a machine and the difference between a machine and human.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringSpatial ReasoningSimilar Papers 제목 키워드 기반
Are LLMs the Master of All Trades? : Exploring Domain-Agnostic Reasoning Skills of LLMs
The potential of large language models (LLMs) to reason like humans has been a highly contested topic in Machine Learning communities. However, the reasoning abilities of humans are multifaceted and can be seen in variou…
AllSpatial Reasoning11Plus-Bench: Demystifying Multimodal LLM Spatial Reasoning with Cognitive-Inspired Analysis
For human cognitive process, spatial reasoning and perception are closely entangled, yet the nature of this interplay remains underexplored in the evaluation of multimodal large language models (MLLMs). While recent MLLM…
Spatial ReasoningAscending the Infinite Ladder: Benchmarking Spatial Deformation Reasoning in Vision-Language Models
Humans naturally possess the spatial reasoning ability to form and manipulate images and structures of objects in space. There is an increasing effort to endow Vision-Language Models (VLMs) with similar spatial reasoning…
Spatial ReasoningBeyond Recognition: Evaluating Visual Perspective Taking in Vision Language Models
We investigate the ability of Vision Language Models (VLMs) to perform visual perspective taking using a novel set of visual tasks inspired by established human tests. Our approach leverages carefully controlled scenes, …
DiagnosticObject RecognitionScene UnderstandingSpatial ReasoningCitySeeker: How Do VLMS Explore Embodied Urban Navigation With Implicit Human Needs?
Vision-Language Models (VLMs) have made significant progress in explicit instruction-based navigation; however, their ability to interpret implicit human needs (e.g., "I am thirsty") in dynamic urban environments remains…
Spatial Reasoning