UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AI
We introduce UnrealZoo, a rich collection of photo-realistic 3D virtual worlds built on Unreal Engine, designed to reflect the complexity and variability of the open worlds. Additionally, we offer a variety of playable entities for embodied AI agents. Based on UnrealCV, we provide a suite of easy-to-use Python APIs and tools for various potential applications, such as data collection, environment augmentation, distributed training, and benchmarking. We optimize the rendering and communication efficiency of UnrealCV to support advanced applications, such as multi-agent interaction. Our experiments benchmark agents in various complex scenes, focusing on visual navigation and tracking, which are fundamental capabilities for embodied visual intelligence. The results yield valuable insights into the advantages of diverse training environments for reinforcement learning (RL) agents and the challenges faced by current embodied vision agents, including those based on RL and large vision-language models (VLMs), in open worlds. These challenges involve latency in closed-loop control in dynamic scenes and reasoning about 3D spatial structures in unstructured terrain.
Code (0)
등록된 구현이 없습니다.
Tasks
BenchmarkingReinforcement Learning (RL)Visual NavigationSimilar Papers 제목 키워드 기반
Virtual Worlds as Proxy for Multi-Object Tracking Analysis
Modern computer vision algorithms typically require expensive data acquisition and accurate manual labeling. In this work, we instead leverage the recent progress in computer graphics to generate fully labeled, dynamic, …
Instance SegmentationMulti-Object TrackingObjectobject-detection+4State of the Art on Neural Rendering
Efficient rendering of photo-realistic virtual worlds is a long standing effort of computer graphics. Modern graphics techniques have succeeded in synthesizing photo-realistic images from hand-crafted scene representatio…
BIG-bench Machine LearningImage GenerationNeural RenderingNovel View SynthesisGSAC: Leveraging Gaussian Splatting for Photorealistic Avatar Creation with Unity Integration
Photorealistic avatars have become essential for immersive applications in virtual reality (VR) and augmented reality (AR), enabling lifelike interactions in areas such as training simulations, telemedicine, and virtual …
3DGSNeRFUnityUnrealCV: Connecting Computer Vision to Unreal Engine
Computer graphics can not only generate synthetic images and ground truth but it also offers the possibility of constructing virtual worlds in which: (i) an agent can perceive, navigate, and take actions guided by AI alg…
NavigatePhysical SimulationsPhoto-realistic Neural Domain Randomization
Synthetic data is a scalable alternative to manual supervision, but it requires overcoming the sim-to-real domain gap. This discrepancy between virtual and real worlds is addressed by two seemingly opposed approaches: im…
Depth EstimationImage GenerationMonocular Depth EstimationNeural Rendering+2