Generative Image as Action Models
Image-generation diffusion models have been fine-tuned to unlock new capabilities such as image-editing and novel view synthesis. Can we similarly unlock image-generation models for visuomotor control? We present GENIMA, a behavior-cloning agent that fine-tunes Stable Diffusion to 'draw joint-actions' as targets on RGB images. These images are fed into a controller that maps the visual targets into a sequence of joint-positions. We study GENIMA on 25 RLBench and 9 real-world manipulation tasks. We find that, by lifting actions into image-space, internet pre-trained diffusion models can generate policies that outperform state-of-the-art visuomotor approaches, especially in robustness to scene perturbations and generalizing to novel objects. Our method is also competitive with 3D agents, despite lacking priors such as depth, keypoints, or motion-planners.
Code (1)
Tasks
Image GenerationRobot ManipulationRobot Manipulation GeneralizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Improving image generative models with human interactions
GANs provide a framework for training generative models which mimic a data distribution. However, in many cases we wish to train these generative models to optimize some auxiliary objective function within the data it ge…
ImageGem: In-the-wild Generative Image Interaction Dataset for Generative Model Personalization
We introduce ImageGem, a dataset for studying generative models that understand fine-grained individual preferences. We posit that a key challenge hindering the development of such a generative model is the lack of in-th…
Image RetrievalDiffusion Recommender Model
Generative models such as Generative Adversarial Networks (GANs) and Variational Auto-Encoders (VAEs) are widely utilized to model the generative process of user interactions. However, these generative models suffer from…
DenoisingImage GenerationmodelRecommendation SystemsHow transfer learning is used in generative models for image classification: improved accuracy
Recent breakthroughs in generative neural networks have paved the way for transformative capabilities, particularly in their capacity to generate novel data, notably in the realm of images. The integration of these model…
ClassificationGenerative Adversarial Networkimage-classificationImage Classification+1EvalGIM: A Library for Evaluating Generative Image Models
As the use of text-to-image generative models increases, so does the adoption of automatic benchmarking methods used in their evaluation. However, while metrics and datasets abound, there are few unified benchmarking lib…
BenchmarkingDiversity