Seeing things or seeing scenes: Investigating the capabilities of V&L models to align scene descriptions to images
Images can be described in terms of the objects they contain, or in terms of the types of scene or place that they instantiate. In this paper we address to what extent pretrained Vision and Language models can learn to align descriptions of both types with images. We compare 3 state-of-the-art models, VisualBERT, LXMERT and CLIP. We find that (i) V\&L models are susceptible to stylistic biases acquired during pretraining; (ii) only CLIP performs consistently well on both object- and scene-level descriptions. A follow-up ablation study shows that CLIP uses object-level information in the visual modality to align with scene-level textual descriptions.
Code (0)
등록된 구현이 없습니다.
Tasks
ObjectMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Are You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on Twitter
Vision: looking and seeing through our brain's information bottleneck
Our brain recognizes only a tiny fraction of sensory input, due to an information processing bottleneck. This blinds us to most visual inputs. Since we are blind to this blindness, only a recent framework highlights this…
Seeing Through Clutter: Structured 3D Scene Reconstruction via Iterative Object Removal
We present SeeingThroughClutter, a method for reconstructing structured 3D representations from single images by segmenting and modeling objects individually. Prior approaches rely on intermediate tasks such as semantic …
Semantic SegmentationDepth EstimationSeeing Faces in Things: A Model and Dataset for Pareidolia
The human visual system is well-tuned to detect faces of all shapes and sizes. While this brings obvious survival advantages, such as a better chance of spotting unknown predators in the bush, it also leads to spurious f…
Single Image 3D Without a Single 3D Image
Do we really need 3D labels in order to learn how to predict 3D? In this paper, we show that one can learn a mapping from appearance to 3D properties without ever seeing a single explicit 3D label. Rather than use explic…
Scene Understanding