Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark
Tracking by natural language specification is a new rising research topic that aims at locating the target object in the video sequence based on its language description. Compared with traditional bounding box (BBox) based tracking, this setting guides object tracking with high-level semantic information, addresses the ambiguity of BBox, and links local and global search organically together. Those benefits may bring more flexible, robust and accurate tracking performance in practical scenarios. However, existing natural language initialized trackers are developed and compared on benchmark datasets proposed for tracking-by-BBox, which can't reflect the true power of tracking-by-language. In this work, we propose a new benchmark specifically dedicated to the tracking-by-language, including a large scale dataset, strong and diverse baseline methods. Specifically, we collect 2k video sequences (contains a total of 1,244,340 frames, 663 words) and split 1300/700 for the train/testing respectively. We densely annotate one sentence in English and corresponding bounding boxes of the target object for each video. We also introduce two new challenges into TNL2K for the object tracking task, i.e., adversarial samples and modality switch. A strong baseline method based on an adaptive local-global-search scheme is proposed for future works to compare. We believe this benchmark will greatly boost related researches on natural language guided tracking.
Code (2)
Tasks
2kObjectObject TrackingSentenceVisual Object TrackingVisual TrackingSimilar Papers 제목 키워드 기반
FoodTrack: Estimating Handheld Food Portions with Egocentric Video
Accurately tracking food consumption is crucial for nutrition and health monitoring. Traditional approaches typically require specific camera angles, non-occluded images, or rely on gesture recognition to estimate intake…
Gesture RecognitionNutritionA Flexible-Frame-Rate Vision-Aided Inertial Object Tracking System for Mobile Devices
Real-time object pose estimation and tracking is challenging but essential for emerging augmented reality (AR) applications. In general, state-of-the-art methods address this problem using deep neural networks which inde…
3D Pose EstimationObjectObject TrackingPose EstimationInterpretable and Robust Dialogue State Tracking via Natural Language Summarization with LLMs
This paper introduces a novel approach to Dialogue State Tracking (DST) that leverages Large Language Models (LLMs) to generate natural language descriptions of dialogue states, moving beyond traditional slot-value repre…
Dialogue State Trackingslot-fillingSlot FillingTask-Oriented Dialogue SystemsExplaining human multiple object tracking as resource-constrained approximate inference in a dynamic probabilistic model
Multiple object tracking is a task commonly used to investigate the architecture of human visual attention. Human participants show a distinctive pattern of successes and failures in tracking experiments that is often at…
Multiple Object TrackingObjectObject TrackingTracking using Numerous Anchor points
In this paper, an online adaptive model-free tracker is proposed to track single objects in video sequences to deal with real-world tracking challenges like low-resolution, object deformation, occlusion and motion blur. …
ObjectObject LocalizationVisual Tracking