Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action Representations

2024-04-26 · Puhao Li, Tengyu Liu, Yuyang Li, Muzhi Han, Haoran Geng, Shu Wang, Yixin Zhu, Song-Chun Zhu, Siyuan Huang

Autonomous robotic systems capable of learning novel manipulation tasks are poised to transform industries from manufacturing to service automation. However, modern methods (e.g., VIP and R3M) still face significant hurdles, notably the domain gap among robotic embodiments and the sparsity of successful task executions within specific action spaces, resulting in misaligned and ambiguous task representations. We introduce Ag2Manip (Agent-Agnostic representations for Manipulation), a framework aimed at surmounting these challenges through two key innovations: a novel agent-agnostic visual representation derived from human manipulation videos, with the specifics of embodiments obscured to enhance generalizability; and an agent-agnostic action representation abstracting a robot's kinematics to a universal agent proxy, emphasizing crucial interactions between end-effector and object. Ag2Manip's empirical validation across simulated benchmarks like FrankaKitchen, ManiSkill, and PartManip shows a 325% increase in performance, achieved without domain-specific demonstrations. Ablation studies underline the essential contributions of the visual and action representations to this success. Extending our evaluations to the real world, Ag2Manip significantly improves imitation learning success rates from 50% to 77.5%, demonstrating its effectiveness and generalizability across both simulated and physical environments.

📄 PDF Abstract BibTeX arXiv:2404.17521

Code (1)

Xiaoyao-Li/Ag2Manip 공식 구현 pytorch

Tasks

Imitation Learning

Methods 이 논문이 사용한 방법론

Golden Queue Managers 설명 없음

Ag2x2: Robust Agent-Agnostic Visual Representations for Zero-Shot Bimanual Manipulation

2025-07-26 · Ziyin Xiong, Yinghan Chen, Puhao Li, Yixin Zhu 외 arxiv

Bimanual manipulation, fundamental to human daily activities, remains a challenging task due to its inherent complexity of coordinated control. Recent advances have enabled zero-shot learning of single-arm manipulation s…

Zero-Shot Learning

Unsupervised Reinforcement Learning for Transferable Manipulation Skill Discovery

2022-04-29 · Daesol Cho, Jigang Kim, H. Jin Kim

Current reinforcement learning (RL) in robotics often experiences difficulty in generalizing to new downstream tasks due to the innate task-specific training paradigm. To alleviate it, unsupervised RL, a framework that p…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Unsupervised Reinforcement Learning

Lifelong Language-Conditioned Robotic Manipulation Learning

2026-03-05 · Xudong Wang, Zebin Han, Zhiyu Liu, Gan Li 외 arxiv

Traditional language-conditioned manipulation agent sequential adaptation to new manipulation skills leads to catastrophic forgetting of old skills, limiting dynamic scene practical deployment. In this paper, we propose …

RoboReact: Agentic Skill Distillation from Generated Egocentric Videos for Generalizable Whole-Body Manipulation

2026-08-04 · Shuliang He, Shuai Wang, Bo Yue, Junchi Teng 외 arxiv

Humanoid robots have the potential to perform dexterous manipulation in human environments, yet acquiring diverse and generalizable skills remains costly due to expensive hardware data collection and labor-intensive anno…

3D Reconstruction

Frame Mining: a Free Lunch for Learning Robotic Manipulation from 3D Point Clouds

2022-10-14 · Minghua Liu, Xuanlin Li, Zhan Ling, Yangyan Li 외

We study how choices of input point cloud coordinate frames impact learning of manipulation skills from 3D point clouds. There exist a variety of coordinate frame choices to normalize captured robot-object-interaction po…

3D Point Cloud Reinforcement Learning Imitation Learning Inductive Bias Reinforcement Learning (RL)+1