Zero-shot Hazard Identification in Autonomous Driving: A Case Study on the COOOL Benchmark
This paper presents our submission to the COOOL competition, a novel benchmark for detecting and classifying out-of-label hazards in autonomous driving. Our approach integrates diverse methods across three core tasks: (i) driver reaction detection, (ii) hazard object identification, and (iii) hazard captioning. We propose kernel-based change point detection on bounding boxes and optical flow dynamics for driver reaction detection to analyze motion patterns. For hazard identification, we combined a naive proximity-based strategy with object classification using a pre-trained ViT model. At last, for hazard captioning, we used the MOLMO vision-language model with tailored prompts to generate precise and context-aware descriptions of rare and low-resolution hazards. The proposed pipeline outperformed the baseline methods by a large margin, reducing the relative error by 33%, and scored 2nd on the final leaderboard consisting of 32 teams.
Code (0)
등록된 구현이 없습니다.
Tasks
Autonomous DrivingChange Point DetectionLanguage ModelingLanguage ModellingOptical Flow EstimationSimilar Papers 제목 키워드 기반
Towards a Multi-Agent Vision-Language System for Zero-Shot Novel Hazardous Object Detection for Autonomous Driving Safety
Detecting anomalous hazards in visual data, particularly in video streams, is a critical challenge in autonomous driving. Existing models often struggle with unpredictable, out-of-label hazards due to their reliance on p…
Anomaly DetectionAutonomous DrivingDenoisingLanguage Modeling+7Zero-Shot Semantic Re-Identification for Autonomous Driving: A VLM Baseline Study
Re-Identification (ReID) in autonomous driving is typically formulated as a visual matching problem, where observations of vehicles, pedestrians, and cyclists are associated across time, frames, or camera views using lea…
Autonomous DrivingMulti-Agent Vulnerability Discovery for Autonomous Driving with Hazard Arbitration Reward
Discovering hazardous scenarios is crucial in testing and further improving driving policies. However, conducting efficient driving policy testing faces two key challenges. On the one hand, the probability of naturally e…
Autonomous DrivingMulti-agent Reinforcement LearningVision-based Navigation of Autonomous Vehicle in Roadway Environments with Unexpected Hazards
Vision-based navigation of autonomous vehicles primarily depends on the Deep Neural Network (DNN) based systems in which the controller obtains input from sensors/detectors, such as cameras and produces a vehicle control…
Autonomous DrivingAutonomous VehiclesNavigateobject-detection+3DriveSafe: A Framework for Risk Detection and Safety Suggestions in Driving Scenarios
Comprehensive situational awareness is essential for autonomous vehicles operating in safety-critical environments, as it enables the identification and mitigation of potential risks. Although recent Multimodal Large Lan…
Autonomous VehiclesScene Understanding