Video Object Segmentation using Teacher-Student Adaptation in a Human Robot Interaction (HRI) Setting
Video object segmentation is an essential task in robot manipulation to facilitate grasping and learning affordances. Incremental learning is important for robotics in unstructured environments, since the total number of objects and their variations can be intractable. Inspired by the children learning process, human robot interaction (HRI) can be utilized to teach robots about the world guided by humans similar to how children learn from a parent or a teacher. A human teacher can show potential objects of interest to the robot, which is able to self adapt to the teaching signal without providing manual segmentation labels. We propose a novel teacher-student learning paradigm to teach robots about their surrounding environment. A two-stream motion and appearance "teacher" network provides pseudo-labels to adapt an appearance "student" network. The student network is able to segment the newly learned objects in other scenes, whether they are static or in motion. We also introduce a carefully designed dataset that serves the proposed HRI setup, denoted as (I)nteractive (V)ideo (O)bject (S)egmentation. Our IVOS dataset contains teaching videos of different objects, and manipulation tasks. Unlike previous datasets, IVOS provides manipulation tasks sequences with segmentation annotation along with the waypoints for the robot trajectories. It also provides segmentation annotation for the different transformations such as translation, scale, planar rotation, and out-of-plane rotation. Our proposed adaptation method outperforms the state-of-the-art on DAVIS and FBMS with 6.8% and 1.2% in F-measure respectively. It improves over the baseline on IVOS dataset with 46.1% and 25.9% in mIoU.
Code (1)
Tasks
Incremental LearningRobot ManipulationSegmentationSemantic SegmentationUnsupervised Video Object SegmentationVideo Object SegmentationVideo Semantic SegmentationSimilar Papers 제목 키워드 기반
Unsupervised Video Domain Adaptation with Masked Pre-Training and Collaborative Self-Training
In this work, we tackle the problem of unsupervised domain adaptation (UDA) for video action recognition. Our approach, which we call UNITE, uses an image teacher model to adapt a video student model to the target domain…
Action RecognitionDomain AdaptationTemporal Action LocalizationUnsupervised Domain AdaptationSlotMatch: Distilling Object-Centric Representations for Unsupervised Video Segmentation
Unsupervised video segmentation is a challenging computer vision task, especially due to the lack of supervisory signals coupled with the complexity of visual scenes. To overcome this challenge, state-of-the-art models b…
Knowledge DistillationVideo SegmentationUnsupervised Video Object Segmentation with Online Adversarial Self-Tuning
The existing unsupervised video object segmentation methods depend heavily on the segmentation model trained offline on a labeled training video set, and cannot well generalize to the test videos from a different dom…
ObjectPseudo LabelSegmentationSemantic Segmentation+3First-frame Supervised Video Polyp Segmentation via Propagative and Semantic Dual-teacher Network
Automatic video polyp segmentation plays a critical role in gastrointestinal cancer screening, but the cost of frameby-frame annotations is prohibitively high. While sparse-frame supervised methods have reduced this burd…
BenchmarkingTransfer LearningVideo Polyp SegmentationOnline Model Distillation for Efficient Video Inference
High-quality computer vision models typically address the problem of understanding the general distribution of real-world images. However, most cameras observe only a very small fraction of this distribution. This offers…
modelSegmentationSemantic SegmentationVideo Object Segmentation+1