Self-Supervised Real-to-Sim Scene Generation
Synthetic data is emerging as a promising solution to the scalability issue of supervised deep learning, especially when real data are difficult to acquire or hard to annotate. Synthetic data generation, however, can itself be prohibitively expensive when domain experts have to manually and painstakingly oversee the process. Moreover, neural networks trained on synthetic data often do not perform well on real data because of the domain gap. To solve these challenges, we propose Sim2SG, a self-supervised automatic scene generation technique for matching the distribution of real data. Importantly, Sim2SG does not require supervision from the real-world dataset, thus making it applicable in situations for which such annotations are difficult to obtain. Sim2SG is designed to bridge both the content and appearance gaps, by matching the content of real data, and by matching the features in the source and target domains. We select scene graph (SG) generation as the downstream task, due to the limited availability of labeled datasets. Experiments demonstrate significant improvements over leading baselines in reducing the domain gap both qualitatively and quantitatively, on several synthetic datasets as well as the real-world KITTI dataset.
Code (0)
등록된 구현이 없습니다.
Tasks
Graph GenerationScene GenerationScene Graph GenerationSynthetic Data GenerationTransfer LearningSimilar Papers 제목 키워드 기반
SPSG: Self-Supervised Photometric Scene Generation from RGB-D Scans
We present SPSG, a novel approach to generate high-quality, colored 3D models of scenes from RGB-D scan observations by learning to infer unobserved scene geometry and color in a self-supervised fashion. Our self-supervi…
3D geometry3D ReconstructionScene GenerationSelf-Supervised 3D Scene Flow Estimation and Motion Prediction using Local Rigidity Prior
In this article, we investigate self-supervised 3D scene flow estimation and class-agnostic motion prediction on point clouds. A realistic scene can be well modeled as a collection of rigidly moving parts, therefore its …
Motion Estimationmotion predictionScene Flow EstimationSelf-Supervised LearningR3DPA: Leveraging 3D Representation Alignment and RGB Pretrained Priors for LiDAR Scene Generation
LiDAR scene synthesis is an emerging solution to scarcity in 3D data for robotic tasks such as autonomous driving. Recent approaches employ diffusion or flow matching models to generate realistic scenes, but 3D data rema…
Autonomous DrivingScene GenerationPoint CloudsSelf-Supervised Relation Alignment for Scene Graph Generation
The goal of scene graph generation is to predict a graph from an input image, where nodes correspond to identified and localized objects and edges to their corresponding interaction predicates. Existing methods are train…
Graph GenerationRelationRelation PredictionScene Graph GenerationRealMonoDepth: Self-Supervised Monocular Depth Estimation for General Scenes
We present a generalised self-supervised learning approach for monocular estimation of the real depth across scenes with diverse depth ranges from 1--100s of meters. Existing supervised methods for monocular depth estima…
Depth EstimationMonocular Depth EstimationSelf-Supervised Learning