RenderWorld: World Model with Self-Supervised 3D Label
End-to-end autonomous driving with vision-only is not only more cost-effective compared to LiDAR-vision fusion but also more reliable than traditional methods. To achieve a economical and robust purely visual autonomous driving system, we propose RenderWorld, a vision-only end-to-end autonomous driving framework, which generates 3D occupancy labels using a self-supervised gaussian-based Img2Occ Module, then encodes the labels by AM-VAE, and uses world model for forecasting and planning. RenderWorld employs Gaussian Splatting to represent 3D scenes and render 2D images greatly improves segmentation accuracy and reduces GPU memory consumption compared with NeRF-based methods. By applying AM-VAE to encode air and non-air separately, RenderWorld achieves more fine-grained scene element representation, leading to state-of-the-art performance in both 4D occupancy forecasting and motion planning from autoregressive world model.
Code (0)
등록된 구현이 없습니다.
Tasks
Autonomous DrivingGPUmodelMotion PlanningNeRFSimilar Papers 제목 키워드 기반
Self-supervised AutoFlow
Recently, AutoFlow has shown promising results on learning a training set for optical flow, but requires ground truth labels in the target domain to compute its search metric. Observing a strong correlation between the g…
Optical Flow EstimationOwMatch: Conditional Self-Labeling with Consistency for Open-World Semi-Supervised Learning
Semi-supervised learning (SSL) offers a robust framework for harnessing the potential of unannotated data. Traditionally, SSL mandates that all classes possess labeled instances. However, the emergence of open-world SSL …
Open-World Semi-Supervised LearningSOHES: Self-supervised Open-world Hierarchical Entity Segmentation
Open-world entity segmentation, as an emerging computer vision task, aims at segmenting entities in images without being restricted by pre-defined classes, offering impressive generalization capabilities on unseen images…
SegmentationSelf-supervised Graph Neural Networks without explicit negative sampling
Real world data is mostly unlabeled or only few instances are labeled. Manually labeling data is a very expensive and daunting task. This calls for unsupervised learning techniques that are powerful enough to achieve com…
Contrastive LearningData AugmentationGraph Neural NetworkSelf-Supervised LearningGraFN: Semi-Supervised Node Classification on Graph with Few Labels via Non-Parametric Distribution Assignment
Despite the success of Graph Neural Networks (GNNs) on various applications, GNNs encounter significant performance degradation when the amount of supervision signals, i.e., number of labeled nodes, is limited, which is …
Node ClassificationSelf-Supervised Learning