Rain rendering for evaluating and improving robustness to bad weather
Rain fills the atmosphere with water particles, which breaks the common assumption that light travels unaltered from the scene to the camera. While it is well-known that rain affects computer vision algorithms, quantifying its impact is difficult. In this context, we present a rain rendering pipeline that enables the systematic evaluation of common computer vision algorithms to controlled amounts of rain. We present three different ways to add synthetic rain to existing images datasets: completely physic-based; completely data-driven; and a combination of both. The physic-based rain augmentation combines a physical particle simulator and accurate rain photometric modeling. We validate our rendering methods with a user study, demonstrating our rain is judged as much as 73% more realistic than the state-of-theart. Using our generated rain-augmented KITTI, Cityscapes, and nuScenes datasets, we conduct a thorough evaluation of object detection, semantic segmentation, and depth estimation algorithms and show that their performance decreases in degraded weather, on the order of 15% for object detection, 60% for semantic segmentation, and 6-fold increase in depth estimation error. Finetuning on our augmented synthetic data results in improvements of 21% on object detection, 37% on semantic segmentation, and 8% on depth estimation.
Code (0)
등록된 구현이 없습니다.
Tasks
Depth EstimationObjectobject-detectionObject DetectionSegmentationSemantic SegmentationSimilar Papers 제목 키워드 기반
Physics-Based Rendering for Improving Robustness to Rain
To improve the robustness to rain, we present a physically-based rain rendering pipeline for realistically inserting rain into clear weather images. Our rendering relies on a physical particle simulator, an estimation of…
Objectobject-detectionObject DetectionSegmentation+1IntrinsicWeather: Controllable Weather Editing in Intrinsic Space
We present IntrinsicWeather, a diffusion-based framework for controllable weather editing in intrinsic space. Our framework includes two components based on diffusion priors: an inverse renderer that estimates material p…
Autonomous DrivingDistracting Downpour: Adversarial Weather Attacks for Motion Estimation
Current adversarial attacks on motion estimation, or optical flow, optimize small per-pixel perturbations, which are unlikely to appear in the real world. In contrast, adverse weather conditions constitute a much more re…
Motion EstimationOptical Flow EstimationInto the Fog: Evaluating Robustness of Multiple Object Tracking
State-of-the-art Multiple Object Tracking (MOT) approaches have shown remarkable performance when trained and evaluated on current benchmarks. However, these benchmarks primarily consist of clear weather scenarios, overl…
Depth EstimationMonocular Depth EstimationMultiple Object TrackingObject+1Weather-Magician: Reconstruction and Rendering Framework for 4D Weather Synthesis In Real Time
For tasks such as urban digital twins, VR/AR/game scene design, or creating synthetic films, the traditional industrial approach often involves manually modeling scenes and using various rendering engines to complete the…