Contrast Sets for Evaluating Language-Guided Robot Policies
Robot evaluations in language-guided, real world settings are time-consuming and often sample only a small space of potential instructions across complex scenes. In this work, we introduce contrast sets for robotics as an approach to make small, but specific, perturbations to otherwise independent, identically distributed (i.i.d.) test instances. We investigate the relationship between experimenter effort to carry out an evaluation and the resulting estimated test performance as well as the insights that can be drawn from performance on perturbed instances. We use the relative performance change of different contrast set perturbations to characterize policies at reduced experimenter effort in both a simulated manipulation task and a physical robot vision-and-language navigation task. We encourage the use of contrast set evaluations as a more informative alternative to small scale, i.i.d. demonstrations on physical robots, and as a scalable alternative to industry-scale real world evaluations.
Code (0)
등록된 구현이 없습니다.
Tasks
Vision and Language NavigationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Contrastive Imitation Learning for Language-guided Multi-Task Robotic Manipulation
Developing robots capable of executing various manipulation tasks, guided by natural language instructions and visual observations of intricate real-world environments, remains a significant challenge in robotics. Such r…
Imitation LearningLanguage-Driven Representation Learning for Robotics
Recent work in visual representation learning for robotics demonstrates the viability of learning from large video datasets of humans performing everyday tasks. Leveraging methods such as masked autoencoding and contrast…
Contrastive LearningImitation LearningRepresentation LearningText GenerationA Taxonomy for Evaluating Generalist Robot Policies
Machine learning for robotics promises to unlock generalization to novel tasks and environments. Guided by this promise, many recent works have focused on scaling up robot data collection and developing larger, more expr…
Robot ManipulationVision-Language-ActionKiloBot: A Programming Language for Deploying Perception-Guided Industrial Manipulators at Scale
We would like industrial robots to handle unstructured environments with cameras and perception pipelines. In contrast to traditional industrial robots that replay offline-crafted trajectories, online behavior planning i…
Industrial RobotsMotion PlanningTask and Motion PlanningAdaptive Language-Guided Abstraction from Contrastive Explanations
Many approaches to robot learning begin by inferring a reward function from a set of human demonstrations. To learn a good reward, it is necessary to determine which features of the environment are relevant before determ…
Program Synthesis