KV-Tracker: Real-Time Pose Tracking with Transformers
Multi-view 3D geometry networks offer a powerful prior but are prohibitively slow for real-time applications. We propose a novel way to adapt them for online use, enabling real-time 6-DoF pose tracking and online reconstruction of objects and scenes from monocular RGB videos. Our method rapidly selects and manages a set of images as keyframes to map a scene or object via $π^3$ with full bidirectional attention. We then cache the global self-attention block's key-value (KV) pairs and use them as the sole scene representation for online tracking. This allows for up to $15\times$ speedup during inference without the fear of drift or catastrophic forgetting. Our caching strategy is model-agnostic and can be applied to other off-the-shelf multi-view networks without retraining. We demonstrate KV-Tracker on both scene-level tracking and the more challenging task of on-the-fly object tracking and reconstruction without depth measurements or object priors. Experiments on the TUM RGB-D, 7-Scenes, Arctic and OnePose datasets show the strong performance of our system while maintaining high frame-rates up to ${\sim}27$ FPS.
Code (0)
등록된 구현이 없습니다.
Tasks
Object TrackingPose TrackingSimilar Papers 제목 키워드 기반
Efficient Visual Tracking with Exemplar Transformers
The design of more complex and powerful neural network models has significantly advanced the state-of-the-art in visual object tracking. These advances can be attributed to deeper networks, or the introduction of new bui…
CPUObject TrackingVideo Object TrackingVisual Object Tracking+1Mobile Vision Transformer-based Visual Object Tracking
The introduction of robust backbones, such as Vision Transformers, has improved the performance of object tracking algorithms in recent years. However, these state-of-the-art trackers are computationally expensive since …
GPUObjectObject TrackingVisual Object TrackingSeparable Self and Mixed Attention Transformers for Efficient Object Tracking
The deployment of transformers for visual object tracking has shown state-of-the-art results on several benchmarks. However, the transformer-based models are under-utilized for Siamese lightweight tracking due to the com…
CPUGPUObject TrackingState Estimation+1Lightweight RGB-T Tracking with Mobile Vision Transformers
Single-modality object tracking (e.g., RGB-only) encounters difficulties in challenging imaging conditions, such as low illumination and adverse weather conditions. To solve this, multimodal tracking (e.g., RGB-T models)…
GPUObject TrackingRgb-T TrackingOn Designing Light-Weight Object Trackers through Network Pruning: Use CNNs or Transformers?
Object trackers deployed on low-power devices need to be light-weight, however, most of the current state-of-the-art (SOTA) methods rely on using compute-heavy backbones built using CNNs or transformers. Large sizes of s…
Network PruningObjectObject Tracking