Sim2real Image Translation Enables Viewpoint-Robust Policies from Fixed-Camera Datasets

2026-01-14 · Jeremiah Coholich, Justin Wit, Robert Azarcon, Zsolt Kira arxiv

Vision-based policies for robot manipulation have achieved significant recent success, but are still brittle to distribution shifts such as camera viewpoint variations. Robot demonstration data is scarce and often lacks appropriate variation in camera viewpoints. Simulation offers a way to collect robot demonstrations at scale with comprehensive coverage of different viewpoints, but presents a visual sim2real challenge. To bridge this gap, we propose MANGO -- an unpaired image translation method with a novel segmentation-conditioned InfoNCE loss, a highly-regularized discriminator design, and a modified PatchNCE loss. We find that these elements are crucial for maintaining viewpoint consistency during sim2real translation. When training MANGO, we only require a small amount of fixed-camera data from the real world, but show that our method can generate diverse unseen viewpoints by translating simulated observations. In this setting, MANGO outperforms all other image translation methods we tested. In certain real-world tabletop manipulation tasks, MANGO augmentation increases shifted-view success rates by over 40 percentage points compared to policies trained without augmentation.

📄 PDF Abstract BibTeX arXiv:2601.09605

Code (0)

등록된 구현이 없습니다.

Tasks

Robot Manipulation

Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image Translation

2021-06-16 · Alex Church, John Lloyd, Raia Hadsell, Nathan F. Lepora

Simulation has recently become key for deep reinforcement learning to safely and efficiently acquire general and complex control policies from visual and proprioceptive inputs. Tactile information is not usually consider…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

View-Invariant Policy Learning via Zero-Shot Novel View Synthesis

2024-09-05 · Stephen Tian, Blake Wulfe, Kyle Sargent, Katherine Liu 외

Large-scale visuomotor policy learning is a promising approach toward developing generalizable manipulation systems. Yet, policies that can be deployed on diverse embodiments, environments, and observational modalities r…

Data Augmentation Novel View Synthesis

Deep Latent Competition: Learning to Race Using Visual Control Policies in Latent Space

2021-02-19 · Wilko Schwarting, Tim Seyde, Igor Gilitschenski, Lucas Liebenwein 외

Learning competitive behaviors in multi-agent settings such as racing requires long-term reasoning about potential adversarial interactions. This paper presents Deep Latent Competition (DLC), a novel reinforcement learni…

Reinforcement Learning (RL)

Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation

2017-07-11 · YuXuan Liu, Abhishek Gupta, Pieter Abbeel, Sergey Levine

Imitation learning is an effective approach for autonomous systems to acquire control policies when an explicit reward function is unavailable, using supervision provided as demonstrations from an expert, typically a hum…

Deep Reinforcement Learning Imitation Learning Reinforcement Learning Translation+1

Augmenting Imitation Experience via Equivariant Representations

2021-10-14 · Dhruv Sharma, Alihusein Kuwajerwala, Florian Shkurti

The robustness of visual navigation policies trained through imitation often hinges on the augmentation of the training image-action pairs. Traditionally, this has been done by collecting data from multiple cameras, by u…

Data Augmentation Visual Navigation