Application of Vision-Language Model to Pedestrians Behavior and Scene Understanding in Autonomous Driving
Autonomous driving (AD) has experienced significant improvements in recent years and achieved promising 3D detection, classification, and localization results. However, many challenges remain, e.g. semantic understanding of pedestrians' behaviors, and downstream handling for pedestrian interactions. Recent studies in applications of Large Language Models (LLM) and Vision-Language Models (VLM) have achieved promising results in scene understanding and high-level maneuver planning in diverse traffic scenarios. However, deploying the billion-parameter LLMs to vehicles requires significant computation and memory resources. In this paper, we analyzed effective knowledge distillation of semantic labels to smaller Vision networks, which can be used for the semantic representation of complex scenes for downstream decision-making for planning and control.
Code (0)
등록된 구현이 없습니다.
Tasks
Autonomous DrivingDecision MakingKnowledge DistillationLanguage ModelingLanguage ModellingScene UnderstandingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
PSI: A Pedestrian Behavior Dataset for Socially Intelligent Autonomous Car
Prediction of pedestrian behavior is critical for fully autonomous vehicles to drive in busy city streets safely and efficiently. The future autonomous cars need to fit into mixed conditions with not only technical but a…
Autonomous VehiclesCapturing the Dynamics of Pedestrian Traffic Using a Machine Vision System
We developed a machine vision system to automatically capture the dynamics of pedestrians under four different traffic scenarios. By considering the overhead view of each pedestrian as a digital object, the system proces…
ObjectPositionIntend-Wait-Perceive-Cross: Exploring the Effects of Perceptual Limitations on Pedestrian Decision-Making
Current research on pedestrian behavior understanding focuses on the dynamics of pedestrians and makes strong assumptions about their perceptual abilities. For instance, it is often presumed that pedestrians have omnidir…
Decision MakingUnderstanding Pedestrian Behaviors From Stationary Crowd Groups
Pedestrian behavior modeling and analysis is important for crowd scene understanding and has various applications in video surveillance. Stationary crowd groups are a key factor influencing pedestrian walking patterns bu…
Event DetectionScene UnderstandingHeterogeneous Vision-Language Ensemble with Disagreement-Aware Reranking for Text-Based Person Anomaly Retrieval
Text-based person anomaly retrieval aims to retrieve pedestrians exhibiting anomalous behaviors from a large image gallery using natural language descriptions. Compared with conventional text-based person retrieval, this…
Multimodal ReasoningPerson Retrieval