Versatile and Generalizable Manipulation via Goal-Conditioned Reinforcement Learning with Grounded Object Detection

2025-07-14 · Huiyi Wang, Fahim Shahriar, Alireza Azimi, Gautham Vasan, Rupam Mahmood, Colin Bellinger arxiv

General-purpose robotic manipulation, including reach and grasp, is essential for deployment into households and workspaces involving diverse and evolving tasks. Recent advances propose using large pre-trained models, such as Large Language Models and object detectors, to boost robotic perception in reinforcement learning. These models, trained on large datasets via self-supervised learning, can process text prompts and identify diverse objects in scenes, an invaluable skill in RL where learning object interaction is resource-intensive. This study demonstrates how to integrate such models into Goal-Conditioned Reinforcement Learning to enable general and versatile robotic reach and grasp capabilities. We use a pre-trained object detection model to enable the agent to identify the object from a text prompt and generate a mask for goal conditioning. Mask-based goal conditioning provides object-agnostic cues, improving feature sharing and generalization. The effectiveness of the proposed framework is demonstrated in a simulated reach-and-grasp task, where the mask-based goal conditioning consistently maintains a $\sim$90\% success rate in grasping both in and out-of-distribution objects, while also ensuring faster convergence to higher returns.

📄 PDF Abstract BibTeX arXiv:2507.10814

Code (0)

등록된 구현이 없습니다.

Tasks

Self-Supervised Learning Reinforcement Learning Object Detection

Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning

2022-04-27 · Philippe Hansen-Estruch, Amy Zhang, Ashvin Nair, Patrick Yin 외

Building generalizable goal-conditioned agents from rich observations is a key to reinforcement learning (RL) solving real world problems. Traditionally in goal-conditioned RL, an agent is provided with the exact goal th…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

SkillBlender: Towards Versatile Humanoid Whole-Body Loco-Manipulation via Skill Blending

2025-06-11 · Yuxuan Kuang, Haoran Geng, Amine Elhafsi, Tan-Dzung Do 외

Humanoid robots hold significant potential in accomplishing daily tasks across diverse environments thanks to their flexibility and human-like morphology. Recent works have made significant progress in humanoid whole-bod…

Hierarchical Reinforcement Learning Humanoid Control

Physics-informed Goal-Conditioned Reinforcement Learning under Hybrid Contact Dynamics

2026-05-28 · Vittorio Giammarino, Anastasios Manganaris, Ahmed H. Qureshi arxiv

Learning to reach arbitrary goals from sparse feedback requires agents to infer a rich notion of reachability across state--goal pairs. Goal-conditioned reinforcement learning (GCRL) tackles this challenge by learning po…

Reinforcement Learning

Swapped goal-conditioned offline reinforcement learning

2023-02-17 · Wenyan Yang, Huiling Wang, Dingding Cai, Joni Pajarinen 외

Offline goal-conditioned reinforcement learning (GCRL) can be challenging due to overfitting to the given dataset. To generalize agents' skills outside the given dataset, we propose a goal-swapping procedure that generat…

Offline RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Toward Deployable Multi-Robot Collaboration via a Symbolically-Guided Decision Transformer

2025-08-19 · Rathnam Vidushika Rasanji, Jin Wei-Kocsis, Jiansong Zhang, Dongming Gan 외 arxiv

Reinforcement learning (RL) has demonstrated great potential in robotic operations. However, its data-intensive nature and reliance on the Markov Decision Process (MDP) assumption limit its practical deployment in real-w…

Reinforcement Learning Robot Manipulation Decision Making