Self-supervised Learning by View Synthesis
We present view-synthesis autoencoders (VSA) in this paper, which is a self-supervised learning framework designed for vision transformers. Different from traditional 2D pretraining methods, VSA can be pre-trained with multi-view data. In each iteration, the input to VSA is one view (or multiple views) of a 3D object and the output is a synthesized image in another target pose. The decoder of VSA has several cross-attention blocks, which use the source view as value, source pose as key, and target pose as query. They achieve cross-attention to synthesize the target view. This simple approach realizes large-angle view synthesis and learns spatial invariant representation, where the latter is decent initialization for transformers on downstream tasks, such as 3D classification on ModelNet40, ShapeNet Core55, and ScanObjectNN. VSA outperforms existing methods significantly for linear probing and is competitive for fine-tuning. The code will be made publicly available.
Code (0)
등록된 구현이 없습니다.
Tasks
3D ClassificationDecoderSelf-Supervised LearningSimilar Papers 제목 키워드 기반
Self-supervised Light Field View Synthesis Using Cycle Consistency
High angular resolution is advantageous for practical applications of light fields. In order to enhance the angular resolution of light fields, view synthesis methods can be utilized to generate dense intermediate views …
Test-Time Adaptation for Keypoint-Based Spacecraft Pose Estimation Based on Predicted-View Synthesis
Due to the difficulty of replicating the real conditions during training, supervised algorithms for spacecraft pose estimation experience a drop in performance when trained on synthetic data and applied to real operation…
Image GenerationPose EstimationSelf-Supervised LearningSpacecraft Pose Estimation+1RayZer: A Self-supervised Large View Synthesis Model
We present RayZer, a self-supervised multi-view 3D Vision model trained without any 3D supervision, i.e., camera poses and scene geometry, while exhibiting emerging 3D awareness. Concretely, RayZer takes unposed and unca…
modelNovel View SynthesisStructNeRF: Neural Radiance Fields for Indoor Scenes with Structural Hints
Neural Radiance Fields (NeRF) achieve photo-realistic view synthesis with densely captured input images. However, the geometry of NeRF is extremely under-constrained given sparse views, resulting in significant degradati…
Depth EstimationNeRFNovel View SynthesisSpherical View Synthesis for Self-Supervised 360 Depth Estimation
Learning based approaches for depth perception are limited by the availability of clean training data. This has led to the utilization of view synthesis as an indirect objective for learning depth estimation using effici…
3D Depth EstimationDepth Estimation