Geometry without Position? When Positional Embeddings Help and Hurt Spatial Reasoning
This paper revisits the role of positional embeddings (PEs) within vision transformers (ViTs) from a geometric perspective. We show that PEs are not mere token indices but effectively function as geometric priors that shape the spatial structure of the representation. We introduce token-level diagnostics that measure how multi-view geometric consistency in ViT representation depends on consitent PEs. Through extensive experiments on 14 foundation ViT models, we reveal how PEs influence multi-view geometry and spatial reasoning. Our findings clarify the role of PEs as a causal mechanism that governs spatial structure in ViT representations. Our code is provided in https://github.com/shijianjian/vit-geometry-probes
Code (0)
등록된 구현이 없습니다.
Tasks
Spatial ReasoningSimilar Papers 제목 키워드 기반
CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings
Without positional information, attention-based Transformer neural networks are permutation-invariant. Absolute or relative positional embeddings are the most popular ways to feed Transformer models with positional infor…
Machine Translationspeech-recognitionSpeech RecognitionTranslationAitchison Embeddings for Learning Compositional Graph Representations
Representation learning is central to graph machine learning, powering tasks such as link prediction and node classification. However, most graph embeddings are hard to interpret, offering limited insight into how learne…
Representation LearningNode ClassificationLink PredictionGraph EmbeddingPosition Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings
Transformers with causal attention can solve tasks that require positional information without using positional encodings. In this work, we propose and investigate a new hypothesis about how positional information can be…
PositionNot Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models
Vision-Language Models (VLMs) learn a shared feature space for text and images, enabling the comparison of inputs of different modalities. While prior works demonstrated that VLMs organize natural language representation…
Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings
So far, expensive finetuning beyond the pretraining sequence length has been a requirement for effectively extending the context of language models (LM). In this work, we break this key bottleneck by Dropping the Positio…