HOIST-Former: Hand-held Objects Identification Segmentation and Tracking in the Wild
We address the challenging task of identifying segmenting and tracking hand-held objects which is crucial for applications such as human action segmentation and performance evaluation. This task is particularly challenging due to heavy occlusion rapid motion and the transitory nature of objects being hand-held where an object may be held released and subsequently picked up again. To tackle these challenges we have developed a novel transformer-based architecture called HOIST-Former. HOIST-Former is adept at spatially and temporally segmenting hands and objects by iteratively pooling features from each other ensuring that the processes of identification segmentation and tracking of hand-held objects depend on the hands' positions and their contextual appearance. We further refine HOIST-Former with a contact loss that focuses on areas where hands are in contact with objects. Moreover we also contribute an in-the-wild video dataset called HOIST which comprises 4125 videos complete with bounding boxes segmentation masks and tracking IDs for hand-held objects. Through experiments on the HOIST dataset and two additional public datasets we demonstrate the efficacy of HOIST-Former in segmenting and tracking hand-held objects.
Code (0)
등록된 구현이 없습니다.
Tasks
Action SegmentationSegmentationSimilar Papers 제목 키워드 기반
HOIST-Former: Hand-held Objects Identification, Segmentation, and Tracking in the Wild
We address the challenging task of identifying, segmenting, and tracking hand-held objects, which is crucial for applications such as human action segmentation and performance evaluation. This task is particularly challe…
Action SegmentationSegmentationHOIST: Humanoid Optimization with Imitation and Sample-efficient Tuning for Manipulating Suspended Loads
Manipulating suspended payloads with humanoid robots is challenging because the robot can only influence an underactuated, oscillatory load through whole-body motion and intermittent contact. Imitation learning provides …
Reinforcement LearningHORT: Monocular Hand-held Objects Reconstruction with Transformers
Reconstructing hand-held objects in 3D from monocular images remains a significant challenge in computer vision. Most existing approaches rely on implicit 3D representations, which produce overly smooth reconstructions a…
DenoisingLearning Explicit Contact for Implicit Reconstruction of Hand-held Objects from Monocular Images
Reconstructing hand-held objects from monocular RGB images is an appealing yet challenging task. In this task, contacts between hands and objects provide important cues for recovering the 3D geometry of the hand-held obj…
3D geometryObjectImproving and Evaluating Hand-Object Interaction Detection
Understanding hands and the objects they interact with, both directly and through tools, is a key step for tasks ranging from action perception to 3D reconstruction and robotics. Our paper provides several contributions …
3D Reconstruction