Ego3DT: Tracking Every 3D Object in Ego-centric Videos
The growing interest in embodied intelligence has brought ego-centric perspectives to contemporary research. One significant challenge within this realm is the accurate localization and tracking of objects in ego-centric videos, primarily due to the substantial variability in viewing angles. Addressing this issue, this paper introduces a novel zero-shot approach for the 3D reconstruction and tracking of all objects from the ego-centric video. We present Ego3DT, a novel framework that initially identifies and extracts detection and segmentation information of objects within the ego environment. Utilizing information from adjacent video frames, Ego3DT dynamically constructs a 3D scene of the ego view using a pre-trained 3D scene reconstruction model. Additionally, we have innovated a dynamic hierarchical association mechanism for creating stable 3D tracking trajectories of objects in ego-centric videos. Moreover, the efficacy of our approach is corroborated by extensive experiments on two newly compiled datasets, with 1.04x - 2.90x in HOTA, showcasing the robustness and accuracy of our method in diverse ego-centric scenarios.
Code (0)
등록된 구현이 없습니다.
Tasks
3D Reconstruction3D Scene ReconstructionSimilar Papers 제목 키워드 기반
Tracking Multiple Deformable Objects in Egocentric Videos
Most existing multiple object tracking (MOT) methods that solely rely on appearance features struggle in tracking highly deformable objects. Other MOT methods that use motion clues to associate identities across fram…
DisentanglementMotion DisentanglementMulti-Object TrackingMultiple Object Tracking+1Track4World: Feedforward World-centric Dense 3D Tracking of All Pixels
Estimating the 3D trajectory of every pixel from a monocular video is crucial and promising for a comprehensive understanding of the 3D dynamics of videos. Recent monocular 3D tracking works demonstrate impressive perfor…
Instance Tracking in 3D Scenes from Egocentric Videos
Egocentric sensors such as AR/VR devices capture human-object interactions and offer the potential to provide task-assistance by recalling 3D locations of objects of interest in the surrounding environment. This capabili…
Human-Object Interaction DetectionObject Tracking3D-Aware Instance Segmentation and Tracking in Egocentric Videos
Egocentric videos present unique challenges for 3D scene understanding due to rapid camera motion, frequent object occlusions, and limited object visibility. This paper introduces a novel approach to instance segmentatio…
3D Object ReconstructionInstance SegmentationObjectObject Reconstruction+5VOVTrack: Exploring the Potentiality in Videos for Open-Vocabulary Object Tracking
Open-vocabulary multi-object tracking (OVMOT) represents a critical new challenge involving the detection and tracking of diverse object categories in videos, encompassing both seen categories (base classes) and unseen c…
Multi-Object TrackingObjectobject-detectionObject Detection+4