Not all Views are Created Equal: Analyzing Viewpoint Instabilities in Vision Foundation Models
In this paper, we analyze the viewpoint stability of foundational models - specifically, their sensitivity to changes in viewpoint- and define instability as significant feature variations resulting from minor changes in viewing angle, leading to generalization gaps in 3D reasoning tasks. We investigate nine foundational models, focusing on their responses to viewpoint changes, including the often-overlooked accidental viewpoints where specific camera orientations obscure an object's true 3D structure. Our methodology enables recognizing and classifying out-of-distribution (OOD), accidental, and stable viewpoints using feature representations alone, without accessing the actual images. Our findings indicate that while foundation models consistently encode accidental viewpoints, they vary in their interpretation of OOD viewpoints due to inherent biases, at times leading to object misclassifications based on geometric resemblance. Through quantitative and qualitative evaluations on three downstream tasks - classification, VQA, and 3D reconstruction - we illustrate the impact of viewpoint instability and underscore the importance of feature robustness across diverse viewing conditions.
Code (0)
등록된 구현이 없습니다.
Tasks
3D ReconstructionAllVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
Are Hyperbolic Representations in Graphs Created Equal?
Recently there was an increasing interest in applications of graph neural networks in non-Euclidean geometry; however, are non-Euclidean representations always useful for graph learning tasks? For different problems such…
Graph LearningGraph Representation LearningLink PredictionNode Classification+1Analyzing 3D Objects in Cluttered Images
We present an approach to detecting and analyzing the 3D configuration of objects in real-world images with heavy occlusion and clutter. We focus on the application of finding and analyzing cars. We do so with a two-stag…
3D Shape ReconstructionViewpoint EstimationImproving Person Re-Identification via Pose-Aware Multi-Shot Matching
Person re-identification is the problem of recognizing people across images or videos from non-overlapping views. Although there has been much progress in person re-identification for the last decade, it still remains a …
Person Re-IdentificationPaMM: Pose-aware Multi-shot Matching for Improving Person Re-identification
Person re-identification is the problem of recognizing people across different images or videos with non-overlapping views. Although there has been much progress in person re-identification over the last decade, it remai…
Person Re-IdentificationLocal Intrinsic Dimension Unveils Hallucinations in Diffusion Models
Diffusion models are prone to generating structural hallucinations - samples that match the statistical properties of the training data yet defy underlying structural rules, resulting in anomalies like hands with more th…