paper-with-me

Papers

LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs

2023-12-07 · CVPR 2024 1 · Yunsheng Ma, Can Cui, Xu Cao, Wenqian Ye, Peiran Liu, Juanwu Lu, Amr Abdelraouf, Rohit Gupta, Kyungtae Han, Aniket Bera, James M. Rehg, Ziran Wang

Autonomous driving (AD) has made significant strides in recent years. However, existing frameworks struggle to interpret and execute spontaneous user instructions, such as "overtake the car ahead." Large Language Models (LLMs) have demonstrated impressive reasoning capabilities showing potential to bridge this gap. In this paper, we present LaMPilot, a novel framework that integrates LLMs into AD systems, enabling them to follow user instructions by generating code that leverages established functional primitives. We also introduce LaMPilot-Bench, the first benchmark dataset specifically designed to quantitatively evaluate the efficacy of language model programs in AD. Adopting the LaMPilot framework, we conduct extensive experiments to assess the performance of off-the-shelf LLMs on LaMPilot-Bench. Our results demonstrate the potential of LLMs in handling diverse driving scenarios and following user instructions in driving. To facilitate further research in this area, we release our code and data at https://github.com/PurdueDigitalTwin/LaMPilot.

📄 PDF Abstract BibTeX arXiv:2312.04372

Code (1)

purduedigitaltwin/lampilot 공식 구현

Tasks

Autonomous DrivingCode GenerationLanguage ModelingLanguage Modelling

Methods 이 논문이 사용한 방법론

Attention 설명 없음
Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.
Residual Connection 설명 없음
Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…
Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…
Multi-Head Attention 설명 없음
BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…
Adam 설명 없음

Similar Papers 제목 키워드 기반

OpenAD: Open-World Autonomous Driving Benchmark for 3D Object Detection

2024-11-26 · Zhongyu Xia, Jishuo Li, Zhiwei Lin, Xinhao Wang 외

Open-world autonomous driving encompasses domain generalization and open-vocabulary. Domain generalization refers to the capabilities of autonomous driving systems across different scenarios and sensor parameter configur…

3D Object DetectionAutonomous DrivingDomain GeneralizationLanguage Modeling+6

COOOL: Challenge Of Out-Of-Label A Novel Benchmark for Autonomous Driving

2024-12-06 · Ali K. AlShami, Ananya Kalita, Ryan Rabinowitz, Khang Lam 외

As the Computer Vision community rapidly develops and advances algorithms for autonomous driving systems, the goal of safer and more efficient autonomous transportation is becoming increasingly achievable. However, it is…

Anomaly DetectionAutonomous DrivingDomain AdaptationOpen Set Learning+1

OpenSatMap: A Fine-grained High-resolution Satellite Dataset for Large-scale Map Construction

2024-10-30 · Hongbo Zhao, Lue Fan, Yuntao Chen, Haochen Wang 외

In this paper, we propose OpenSatMap, a fine-grained, high-resolution satellite dataset for large-scale map construction. Map construction is one of the foundations of the transportation industry, such as navigation and …

Autonomous DrivingDiversity

DriveLMM-o1: A Step-by-Step Reasoning Dataset and Large Multimodal Model for Driving Scenario Understanding

2025-03-13 · Ayesha Ishaq, Jean Lahoud, Ketan More, Omkar Thawakar 외

While large multimodal models (LMMs) have demonstrated strong performance across various Visual Question Answering (VQA) tasks, certain challenges require complex multi-step reasoning to reach accurate answers. One parti…

4kAutonomous DrivingQuestion AnsweringVisual Question Answering+2

OpenREAD: Reinforced Open-Ended Reasoning for End-to-End Autonomous Driving with LLM-as-Critic

2025-12-01 · Songyan Zhang, Wenhui Huang, Zhan Chen, Chua Jiahao Collister 외 arxiv

Recently, two-stage fine-tuning strategies, e.g., acquiring essential driving knowledge through supervised fine-tuning (SFT) and further enhancing decision-making and planning via reinforcement fine-tuning (RFT), have sh…

Scene UnderstandingTrajectory PlanningAutonomous Driving