CoDex: Learning Compositional Dexterous Functional Manipulation without Demonstrations

2026-06-30 · Bowen Jiang, William Painter Reger, Roberto Martin-Martin arxiv

In this work, we study Compositional Dexterous Functional Object Manipulation (CD-FOM): tasks such as aiming and actuating a spray bottle on a plant or a glue gun on wood, which require both actuating an object's internal mechanism and controlling its pose to apply the object's function to the environment. These tasks pose significant challenges for robots due to the demanding integration of semantic understanding of the object's function, actuation mode, and application area with intricate physical dexterity to manage grasp stability, movement trajectory, and actuation. We introduce CoDex, a zero-demonstration framework that autonomously discovers CD-FOM manipulation strategies. CoDex uses vision-language models (VLMs) to infer semantic constraints from the task and scene. These constraints guide analytic constrained optimization to generate a short list of functional grasp candidates that can be efficiently refined with reinforcement learning to generate full grasp-move-actuate policies transferable from simulation to the real world. We evaluate CoDex on a 7-DoF robot arm with a 16-DoF multi-fingered hand across six CD-FOM tasks involving previously unseen objects with internal mechanisms, including spray bottles, hot glue guns, air dusters, flashlights, and pepper grinders, and their application to unseen target objects, showcasing its ability to autonomously discover and execute complex, physically viable dexterous behaviors without human demonstrations. More information at https://robin-lab.cs.utexas.edu/CoDex/.

📄 PDF Abstract BibTeX arXiv:2606.31909

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation

2023-09-02 · Yuanpei Chen, Chen Wang, Li Fei-Fei, C. Karen Liu

Many real-world manipulation tasks consist of a series of subtasks that are significantly different from one another. Such long-horizon, complex tasks highlight the potential of dexterous hands, which possess adaptabilit…

Reinforcement Learning (RL)

BLaDA: Bridging Language to Functional Dexterous Actions within 3DGS Fields

2026-04-09 · Fan Yang, Wenrui Chen, Guorun Yan, Ruize Liao 외 arxiv

In unstructured environments, functional dexterous grasping calls for the tight integration of semantic understanding, precise 3D functional localization, and physically interpretable execution. Modular hierarchical meth…

Spatial Reasoning

Mana: Dexterous Manipulation of Articulated Tools

2026-06-11 · Zhao-Heng Yin, Guanya Shi, Pieter Abbeel, C. Karen Liu arxiv

Articulated tool manipulation remains a major challenge in dexterous robotics due to the need to coordinate internal degrees of freedom and contact-rich interactions. While prior work has largely focused on rigid objects…

Reinforcement Learning Motion Planning

HANDFUL: Sequential Grasp-Conditioned Dexterous Manipulation with Resource Awareness

2026-04-28 · Ethan Foong, Yunshuang Li, Hao Jiang, Gaurav S. Sukhatme 외 arxiv

Dexterous robot hands offer rich opportunities for multifunctional manipulation, where a robot must execute multiple skills in sequence while maintaining control over previously grasped objects. Most prior work in dexter…

LabDex: A Hierarchical Benchmark for Dexterous Manipulation in Laboratories

2026-08-19 · Zhipeng Tang, Sihang Chen, Sha Zhang, Peihao Yang 외 arxiv

Autonomous laboratories hold great promise for accelerating scientific discovery. To achieve this vision, robots are supposed to dexterously manipulate diverse labware and instruments and execute long-horizon, state-depe…