Large Language Models for Autonomous Driving (LLM4AD): Concept, Benchmark, Experiments, and Challenges
With the broader usage and highly successful development of Large Language Models (LLMs), there has been a growth of interest and demand for applying LLMs to autonomous driving technology. Driven by their natural language understanding and reasoning ability, LLMs have the potential to enhance various aspects of autonomous driving systems, from perception and scene understanding to language interaction and decision-making. In this paper, we first introduce the novel concept of designing LLMs for autonomous driving (LLM4AD). Then, we propose a comprehensive benchmark for evaluating the instruction-following abilities of LLM4AD in simulation. Furthermore, we conduct a series of experiments on real-world vehicle platforms, thoroughly evaluating the performance and potential of our LLM4AD systems. Finally, we envision the main challenges of LLM4AD, including latency, deployment, security and privacy, safety, trust and transparency, and personalization. Our research highlights the significant potential of LLMs to enhance various aspects of autonomous vehicle technology, from perception and scene understanding to language interaction and decision-making.
Code (0)
등록된 구현이 없습니다.
Tasks
Autonomous DrivingDecision MakingInstruction FollowingNatural Language UnderstandingNovel ConceptsScene UnderstandingSimilar Papers 제목 키워드 기반
A Framework for a Capability-driven Evaluation of Scenario Understanding for Multimodal Large Language Models in Autonomous Driving
Multimodal large language models (MLLMs) hold the potential to enhance autonomous driving by combining domain-independent world knowledge with context-specific language guidance. Their integration into autonomous driving…
Autonomous DrivingDecision MakingWorld KnowledgeA Survey on Multimodal Large Language Models for Autonomous Driving
With the emergence of Large Language Models (LLMs) and Vision Foundation Models (VFMs), multimodal AI systems benefiting from large models have the potential to equally perceive the real world, make decisions, and contro…
Autonomous DrivingEvaluation of Safety Cognition Capability in Vision-Language Models for Autonomous Driving
Assessing the safety of vision-language models (VLMs) in autonomous driving is particularly important; however, existing work mainly focuses on traditional benchmark evaluations. As interactive components within autonomo…
Autonomous Drivingtext annotationEvaluation of Large Language Models for Decision Making in Autonomous Driving
Various methods have been proposed for utilizing Large Language Models (LLMs) in autonomous driving. One strategy of using LLMs for autonomous driving involves inputting surrounding objects as text prompts to the LLMs, a…
Autonomous DrivingDecision MakingTalk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Driving
Talk2BEV is a large vision-language model (LVLM) interface for bird's-eye view (BEV) maps in autonomous driving contexts. While existing perception systems for autonomous driving scenarios have largely focused on a pre-d…
Autonomous DrivingDecision MakingLanguage ModelingLanguage Modelling+3