Bridging the Sim2Real Gap: Vision Encoder Pre-Training for Visuomotor Policy Transfer
Simulation offers a scalable and efficient alternative to real-world data collection for learning visuomotor robotic policies. However, the simulation-to-reality, or "Sim2Real" distribution shift -- introduced by employing simulation-trained policies in real-world environments -- frequently prevents successful policy transfer. This study explores the potential of using large-scale pre-training of vision encoders to address the Sim2Real gap. We examine a diverse collection of encoders, evaluating their ability to (1) extract features necessary for robot control while (2) remaining invariant to task-irrelevant environmental variations. We quantitatively measure the encoder's feature extraction capabilities through linear probing and its domain invariance by computing distances between simulation and real-world embedding centroids. Additional qualitative insights are provided through t-SNE plots and GradCAM saliency maps. Findings suggest that encoders pre-trained on manipulation-specific datasets generally outperform those trained on generic datasets in bridging the Sim2Real gap. https://github.com/yyardi/Bridging-the-Sim2Real-Gap
Code (1)
Similar Papers 제목 키워드 기반
GaussFly: Contrastive Reinforcement Learning for Visuomotor Policies in 3D Gaussian Fields
Learning visuomotor policies for Autonomous Aerial Vehicles (AAVs) relying solely on monocular vision is an attractive yet highly challenging paradigm. Existing end-to-end learning approaches directly map high-dimensiona…
Representation LearningReinforcement LearningCHOP: Counterfactual Human Preference Labels Improve Obstacle Avoidance in Visuomotor Navigation Policies
Visuomotor navigation policies have shown strong perception-action coupling for embodied agents, yet they often struggle with safe navigation and dynamic obstacle avoidance in complex real-world environments. We introduc…
Contrastive Action-Image Pre-training for Visuomotor Control
Existing vision encoders for robotics face a fundamental bottleneck: robotic datasets lack the scale necessary for large-scale pre-training. Prior work circumvents this data scarcity by turning to internet-scale image an…
DIPOLE: Fusing Vision and Geometry for Robust Visuomotor Generalization
Imitation learning has emerged as a crucial approach for acquiring visuomotor skills from demonstrations, where designing effective observation encoders is essential for policy generalization. However, existing methods t…
Policy Pre-training for Autonomous Driving via Self-supervised Geometric Modeling
Witnessing the impressive achievements of pre-training techniques on large-scale data in the field of computer vision and natural language processing, we wonder whether this idea could be adapted in a grab-and-go spirit,…
Autonomous DrivingDecision Making