RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitation
For robots to be useful outside labs and specialized factories we need a way to teach them new useful behaviors quickly. Current approaches lack either the generality to onboard new tasks without task-specific engineering, or else lack the data-efficiency to do so in an amount of time that enables practical use. In this work we explore dense tracking as a representational vehicle to allow faster and more general learning from demonstration. Our approach utilizes Track-Any-Point (TAP) models to isolate the relevant motion in a demonstration, and parameterize a low-level controller to reproduce this motion across changes in the scene configuration. We show this results in robust robot policies that can solve complex object-arrangement tasks such as shape-matching, stacking, and even full path-following tasks such as applying glue and sticking objects together, all from demonstrations that can be collected in minutes.
Code (1)
Tasks
Point TrackingRobot ManipulationSimilar Papers 제목 키워드 기반
Lattice Boltzmann Model for Learning Real-World Pixel Dynamicity
This work proposes the Lattice Boltzmann Model (LBM) to learn real-world pixel dynamicity for visual tracking. LBM decomposes visual representations into dynamic pixel lattices and solves pixel motion states through coll…
Object TrackingVisual TrackingPoint TrackingHigh Pileup Particle Tracking with Object Condensation
Recent work has demonstrated that graph neural networks (GNNs) can match the performance of traditional algorithms for charged particle tracking while improving scalability to meet the computing challenges posed by the H…
Edge ClassificationObjectOnline Dense Point Tracking with Streaming Memory
Dense point tracking is a challenging task requiring the continuous tracking of every point in the initial frame throughout a substantial portion of a video, even in the presence of occlusions. Traditional methods use op…
Optical Flow EstimationPoint TrackingNeural Stochastic Flows: Solver-Free Modelling and Inference for SDE Solutions
Stochastic differential equations (SDEs) are well suited to modelling noisy and irregularly sampled time series found in finance, physics, and machine learning. Traditional approaches require costly numerical solvers to …
MAGIC: Map-Guided Few-Shot Audio-Visual Acoustics Modeling
Few-shot audio-visual acoustics modeling seeks to synthesize the room impulse response in arbitrary locations with few-shot observations. To sufficiently exploit the provided few-shot data for accurate acoustic modeling,…
Decoder