G3Flow: Generative 3D Semantic Flow for Pose-aware and Generalizable Object Manipulation
Recent advances in imitation learning for 3D robotic manipulation have shown promising results with diffusion-based policies. However, achieving human-level dexterity requires seamless integration of geometric precision and semantic understanding. We present G3Flow, a novel framework that constructs real-time semantic flow, a dynamic, object-centric 3D semantic representation by leveraging foundation models. Our approach uniquely combines 3D generative models for digital twin creation, vision foundation models for semantic feature extraction, and robust pose tracking for continuous semantic flow updates. This integration enables complete semantic understanding even under occlusions while eliminating manual annotation requirements. By incorporating semantic flow into diffusion policies, we demonstrate significant improvements in both terminal-constrained manipulation and cross-object generalization. Extensive experiments across five simulation tasks show that G3Flow consistently outperforms existing approaches, achieving up to 68.3% and 50.1% average success rates on terminal-constrained manipulation and cross-object generalization tasks respectively. Our results demonstrate the effectiveness of G3Flow in enhancing real-time dynamic semantic feature understanding for robotic manipulation policies.
Code (1)
Tasks
Imitation LearningObjectPose TrackingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
SeamFlow: Structure-Aware Flow Matching on Edge Probabilities for Artist-Like UV Unwrapping
3D surface cutting and UV unwrapping are fundamental problems in computer graphics. Traditional geometric optimization methods mainly focus on reducing parameterization distortion, but they often overlook visual semantic…
Boundary-aware Decoupled Flow Networks for Realistic Extreme Rescaling
Recently developed generative methods, including invertible rescaling network (IRN) based and generative adversarial network (GAN) based methods, have demonstrated exceptional performance in image rescaling. However, IRN…
Generative Adversarial NetworkImage RescalingSSIMCosmoFlow: Scale-Aware Representation Learning for Cosmology with Flow Matching
Generative machine learning models have been demonstrated to be able to learn low dimensional representations of data that preserve information required for downstream tasks. In this work, we demonstrate that flow matchi…
Synthetic Data GenerationRepresentation LearningImproving Channel Estimation via Multimodal Diffusion Models with Flow Matching
Deep generative models offer a powerful alternative to conventional channel estimation by learning complex channel distributions. By integrating the rich environmental information available in modern sensing-aided networ…
FlowPipe: LLM-Enhanced Conditional Generative Flow Networks for Data Preparation Pipeline Construction
Data preparation pipelines improve data quality in machine learning by transforming raw tables into learning-ready data through sequential cleaning and feature transformation operators. However, automatically constructin…