Hint-AD: Holistically Aligned Interpretability in End-to-End Autonomous Driving
End-to-end architectures in autonomous driving (AD) face a significant challenge in interpretability, impeding human-AI trust. Human-friendly natural language has been explored for tasks such as driving explanation and 3D captioning. However, previous works primarily focused on the paradigm of declarative interpretability, where the natural language interpretations are not grounded in the intermediate outputs of AD systems, making the interpretations only declarative. In contrast, aligned interpretability establishes a connection between language and the intermediate outputs of AD systems. Here we introduce Hint-AD, an integrated AD-language system that generates language aligned with the holistic perception-prediction-planning outputs of the AD model. By incorporating the intermediate outputs and a holistic token mixer sub-network for effective feature adaptation, Hint-AD achieves desirable accuracy, achieving state-of-the-art results in driving language tasks including driving explanation, 3D dense captioning, and command prediction. To facilitate further study on driving explanation task on nuScenes, we also introduce a human-labeled dataset, Nu-X. Codes, dataset, and models will be publicly available.
Code (0)
등록된 구현이 없습니다.
Tasks
3D dense captioningAutonomous DrivingDense CaptioningSimilar Papers 제목 키워드 기반
Hints of Prompt: Enhancing Visual Representation for Multimodal LLMs in Autonomous Driving
In light of the dynamic nature of autonomous driving environments and stringent safety requirements, general MLLMs combined with CLIP alone often struggle to represent driving-specific scenarios accurately, particularly …
Autonomous DrivingMultimodal ReasoningVisual Question Answering (VQA)HCRMP: A LLM-Hinted Contextual Reinforcement Learning Framework for Autonomous Driving
Integrating Large Language Models (LLMs) with Reinforcement Learning (RL) can enhance autonomous driving (AD) performance in complex scenarios. However, current LLM-Dominated RL methods over-rely on LLM outputs, which ar…
Autonomous DrivingHallucinationMotion PlanningReinforcement Learning (RL)3D Part Guided Image Editing for Fine-Grained Object Understanding
Holistically understanding an object with its 3D movable parts is essential for visual models of a robot to interact with the world. For example, only by understanding many possible part dynamics of other vehicles (e.g.,…
Autonomous DrivingInstance SegmentationSemantic SegmentationDarkDriving: A Real-World Day and Night Aligned Dataset for Autonomous Driving in the Dark Environment
The low-light conditions are challenging to the vision-centric perception systems for autonomous driving in the dark environment. In this paper, we propose a new benchmark dataset (named DarkDriving) to investigate the l…
Autonomous DrivingAutonomous Driving with Deep Learning: A Survey of State-of-Art Technologies
Since DARPA Grand Challenges (rural) in 2004/05 and Urban Challenges in 2007, autonomous driving has been the most active field of AI applications. Almost at the same time, deep learning has made breakthrough by several …
3D Object DetectionAutonomous DrivingDeep LearningDepth Estimation+3