RayRoPE: Projective Ray Positional Encoding for Multi-view Attention
We study positional encodings for multi-view transformers that process tokens from a set of posed input images, and seek a mechanism that encodes patches uniquely, allows SE(3)-invariant attention with multi-frequency similarity, and can adapt to the geometry of the underlying 3D scene. We find that prior (absolute or relative) encoding schemes for multi-view attention do not meet these desiderata, and present RayRoPE to address this gap. RayRoPE represents patch positions based on associated rays and computes query-frame projective coordinates to ensure SE(3) invariance. To adapt to scene geometry, RayRoPE predicts (without direct supervision) a per-token depth to obtain its position along the corresponding ray, while also modeling uncertainty and analytically computing the expected positional encoding. We validate our method on the tasks of novel-view synthesis, stereo depth estimation, and feed-forward 3DGS reconstruction. While remaining efficient, RayRoPE consistently improves over alternate position encoding schemes (e.g., 24% relative improvement on LPIPS in RE10K).
Code (0)
등록된 구현이 없습니다.
Tasks
Stereo Depth EstimationSimilar Papers 제목 키워드 기반
Cameras as Relative Positional Encoding
Transformers are increasingly prevalent for multi-view computer vision tasks, where geometric relationships between viewpoints are critical for 3D perception. To leverage these relationships, multi-view transformers must…
Depth EstimationNovel View SynthesisStereo Depth EstimationCRePE: Curved Ray Expectation Positional Encoding for Unified-Camera-Controlled Video Generation
Camera-conditioned video generation requires positional encoding that remains reliable under changes in camera motion, lens configuration, and scene structure. However, existing attention-level camera encodings either pr…
Video GenerationDPPE: Rethinking Camera-Based Positional Encoding for Scaling Multi-View Transformers
The remarkable scalability of Transformers has expanded their application to 3D computer vision, where camera-aware positional encoding is crucial for providing spatial cues in multi-view geometry. Recent advancements ha…
Novel View SynthesisSpan-based Semantic Parsing for Compositional Generalization
Despite the success of sequence-to-sequence (seq2seq) models in semantic parsing, recent work has shown that they fail in compositional generalization, i.e., the ability to generalize to new structures built of component…
Semantic ParsingGeometry-Aware Rotary Position Embedding for Consistent Video World Model
Predictive world models that simulate future observations under explicit camera control are fundamental to interactive AI. Despite rapid advances, current systems lack spatial persistence: they fail to maintain stable sc…