Efficient Transformer-based 3D Object Detection with Dynamic Token Halting
Balancing efficiency and accuracy is a long-standing problem for deploying deep learning models. The trade-off is even more important for real-time safety-critical systems like autonomous vehicles. In this paper, we propose an effective approach for accelerating transformer-based 3D object detectors by dynamically halting tokens at different layers depending on their contribution to the detection task. Although halting a token is a non-differentiable operation, our method allows for differentiable end-to-end learning by leveraging an equivalent differentiable forward-pass. Furthermore, our framework allows halted tokens to be reused to inform the model's predictions through a straightforward token recycling mechanism. Our method significantly improves the Pareto frontier of efficiency versus accuracy when compared with the existing approaches. By halting tokens and increasing model capacity, we are able to improve the baseline model's performance without increasing the model's latency on the Waymo Open Dataset.
Code (0)
등록된 구현이 없습니다.
Tasks
3D Object DetectionAutonomous Vehiclesobject-detectionObject DetectionSimilar Papers 제목 키워드 기반
HaltingVT: Adaptive Token Halting Transformer for Efficient Video Recognition
Action recognition in videos poses a challenge due to its high computational cost, especially for Joint Space-Time video transformers (Joint VT). Despite their effectiveness, the excessive number of tokens in such archit…
Action RecognitionAction Recognition In VideosToken ReductionVideo RecognitionAdaViT: Adaptive Tokens for Efficient Vision Transformer
We introduce A-ViT, a method that adaptively adjusts the inference cost of vision transformer (ViT) for images of different complexity. A-ViT achieves this by automatically reducing the number of tokens in vision transfo…
Efficient ViTsimage-classificationImage ClassificationToken ReductionNon-Halting Queries: Exploiting Fixed Points in LLMs
We introduce a new vulnerability that exploits fixed points in autoregressive models and use it to craft queries that never halt, i.e. an LLM output that does not terminate. More precisely, for what we call non-halting q…
ADEPT: Adaptive Dynamic Early-Exit Process for Transformers
The inference of large language models imposes significant computational workloads, often requiring the processing of billions of parameters. Although early-exit strategies have proven effective in reducing computational…
AdaPonderLM: Gated Pondering Language Models with Token-Wise Adaptive Depth
Test-time scaling via recurrent/iterative Transformers enables large language models to spend more computation at inference, but most pretrained recurrent LMs run a fixed number of iterations, wasting compute on easy tok…