BEVDriver: Leveraging BEV Maps in LLMs for Robust Closed-Loop Driving
Autonomous driving has the potential to set the stage for more efficient future mobility, requiring the research domain to establish trust through safe, reliable and transparent driving. Large Language Models (LLMs) possess reasoning capabilities and natural language understanding, presenting the potential to serve as generalized decision-makers for ego-motion planning that can interact with humans and navigate environments designed for human drivers. While this research avenue is promising, current autonomous driving approaches are challenged by combining 3D spatial grounding and the reasoning and language capabilities of LLMs. We introduce BEVDriver, an LLM-based model for end-to-end closed-loop driving in CARLA that utilizes latent BEV features as perception input. BEVDriver includes a BEV encoder to efficiently process multi-view images and 3D LiDAR point clouds. Within a common latent space, the BEV features are propagated through a Q-Former to align with natural language instructions and passed to the LLM that predicts and plans precise future trajectories while considering navigation instructions and critical scenarios. On the LangAuto benchmark, our model reaches up to 18.9% higher performance on the Driving Score compared to SoTA methods.
Code (1)
Tasks
Autonomous DrivingMotion PlanningNatural Language UnderstandingNavigateMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Enhancing LLM-based Autonomous Driving with Modular Traffic Light and Sign Recognition
Large Language Models (LLMs) are increasingly used for decision-making and planning in autonomous driving, showing promising reasoning capabilities and potential to generalize across diverse traffic situations. However, …
Autonomous DrivingParametrizations of All Stable Closed-loop Responses: From Theory to Neural Network Control Design
The complexity of modern control systems necessitates architectures that achieve high performance while ensuring robust stability, particularly for nonlinear systems. In this work, we tackle the challenge of designing ou…
AllX-Driver: Explainable Autonomous Driving with Vision-Language Models
End-to-end autonomous driving has advanced significantly, offering benefits such as system simplicity and stronger driving performance in both open-loop and closed-loop settings than conventional pipelines. However, exis…
Autonomous DrivingBench2DriveDecision MakingICR-Drive: Instruction Counterfactual Robustness for End-to-End Language-Driven Autonomous Driving
Recent progress in vision-language-action (VLA) models has enabled language-conditioned driving agents to execute natural-language navigation commands in closed-loop simulation, yet standard evaluations largely assume in…
Autonomous DrivingClosing the Loop: A Control-Theoretic Framework for Provably Stable Time Series Forecasting with LLMs
Large Language Models (LLMs) have recently shown exceptional potential in time series forecasting (TSF), leveraging their inherent sequential reasoning capabilities to model complex temporal dynamics. Existing approaches…
Time Series Forecasting