Future Success Prediction in Open-Vocabulary Object Manipulation Tasks Based on End-Effector Trajectories
This study addresses a task designed to predict the future success or failure of open-vocabulary object manipulation. In this task, the model is required to make predictions based on natural language instructions, egocentric view images before manipulation, and the given end-effector trajectories. Conventional methods typically perform success prediction only after the manipulation is executed, limiting their efficiency in executing the entire task sequence. We propose a novel approach that enables the prediction of success or failure by aligning the given trajectories and images with natural language instructions. We introduce Trajectory Encoder to apply learnable weighting to the input trajectories, allowing the model to consider temporal dynamics and interactions between objects and the end effector, improving the model's ability to predict manipulation outcomes accurately. We constructed a dataset based on the RT-1 dataset, a large-scale benchmark for open-vocabulary object manipulation tasks, to evaluate our method. The experimental results show that our method achieved a higher prediction accuracy than baseline approaches.
Code (0)
등록된 구현이 없습니다.
Tasks
ObjectPredictionSimilar Papers 제목 키워드 기반
V3Det Challenge 2024 on Vast Vocabulary and Open Vocabulary Object Detection: Methods and Results
Detecting objects in real-world scenes is a complex task due to various challenges, including the vast range of object categories, and potential encounters with previously unknown or unseen objects. The challenges necess…
Objectobject-detectionObject DetectionOpen-vocabulary object detection+1CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction
Open-vocabulary dense prediction tasks including object detection and image segmentation have been advanced by the success of Contrastive Language-Image Pre-training (CLIP). CLIP models, particularly those incorporating …
image-classificationImage ClassificationImage Segmentationobject-detection+10Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation
Recent open-vocabulary robot mapping methods enrich dense geometric maps with pre-trained visual-language features. While these maps allow for the prediction of point-wise saliency maps when queried for a certain languag…
ObjectRobot NavigationSHOE: Semantic HOI Open-Vocabulary Evaluation Metric
Open-vocabulary human-object interaction (HOI) detection is a step towards building scalable systems that generalize to unseen interactions in real-world scenarios and support grounded multimodal systems that reason abou…
Semantic SimilarityPre-Manipulation Alignment Prediction with Parallel Deep State-Space and Transformer Models
In this work, we address the problem of predicting the future success of open-vocabulary object manipulation tasks. Conventional approaches typically determine success or failure after the action has been carried out. Ho…