PILOT: A Perceptive Integrated Low-level Controller for Loco-manipulation over Unstructured Scenes

2026-01-24 · Xinru Cui, Linxi Feng, Yixuan Zhou, Haoqi Han, Zhe Liu, Hesheng Wang arxiv

Humanoid robots hold great potential for diverse interactions and daily service tasks within human-centered environments, necessitating controllers that seamlessly integrate precise locomotion with dexterous manipulation. However, most existing whole-body controllers lack exteroceptive awareness of the surrounding environment, rendering them insufficient for stable task execution in complex, unstructured scenarios.To address this challenge, we propose PILOT, a unified single-stage reinforcement learning (RL) framework tailored for perceptive loco-manipulation, which synergizes perceptive locomotion and expansive whole-body control within a single policy. To enhance terrain awareness and ensure precise foot placement, we design a cross-modal context encoder that fuses prediction-based proprioceptive features with attention-based perceptive representations. Furthermore, we introduce a Mixture-of-Experts (MoE) policy architecture to coordinate diverse motor skills, facilitating better specialization across distinct motion patterns. Extensive experiments in both simulation and on the physical Unitree G1 humanoid robot validate the efficacy of our framework. PILOT demonstrates superior stability, command tracking precision, and terrain traversability compared to existing baselines. These results highlight its potential to serve as a robust, foundational low-level controller for loco-manipulation in unstructured scenes.

📄 PDF Abstract BibTeX arXiv:2601.17440

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

Learning Perceptive Platform Adaptive Locomotion Controllers for Quadrupedal Robots

2026-06-23 · David Rytz, Kim Tien Ly, Ioannis Havoutis arxiv

Universal quadrupedal locomotion remains limited by the difficulty of integrating perception across diverse robot morphologies. State-of-the-art controllers rely on single-robot training or blind policies that omit real-…

Reinforcement Learning

Learning to Walk by Steering: Perceptive Quadrupedal Locomotion in Dynamic Environments

2022-09-19 · Mingyo Seo, Ryan Gupta, Yifeng Zhu, Alexy Skoutnev 외

We tackle the problem of perceptive locomotion in dynamic environments. In this problem, a quadrupedal robot must exhibit robust and agile walking behaviors in response to environmental clutter and moving obstacles. We p…

Decision Making Imitation Learning Reinforcement Learning (RL)

ViBe: Visual Behavior Adaptation for Perceptive Humanoid Whole-Body Control

2026-09-09 · Lokesh Krishna, Sarvesh Venkatesan, An Zhang, Quan Nguyen arxiv

Motion tracking provides a scalable recipe for humanoid whole-body control. By design, the resulting trackers lack exteroceptive feedback hence reacting to the environment remains the responsibility of a higher-level pla…

parameter-efficient fine-tuning

Deep Whole-body Parkour

2026-01-12 · Ziwen Zhuang, Shaoting Zhu, Mengjie Zhao, Hang Zhao arxiv

Current approaches to humanoid control generally fall into two paradigms: perceptive locomotion, which handles terrain well but is limited to pedal gaits, and general motion tracking, which reproduces complex skills but …

Integrated Task and Motion Planning for Safe Legged Navigation in Partially Observable Environments

2021-10-23 · Abdulaziz Shamsah, Zhaoyuan Gu, Jonas Warnke, Seth Hutchinson 외

This study proposes a hierarchically integrated framework for safe task and motion planning (TAMP) of bipedal locomotion in a partially observable environment with dynamic obstacles and uneven terrain. The high-level tas…

Motion Planning Task and Motion Planning