Situated Multimodal Control of a Mobile Robot: Navigation through a Virtual Environment
We present a new interface for controlling a navigation robot in novel environments using coordinated gesture and language. We use a TurtleBot3 robot with a LIDAR and a camera, an embodied simulation of what the robot has encountered while exploring, and a cross-platform bridge facilitating generic communication. A human partner can deliver instructions to the robot using spoken English and gestures relative to the simulated environment, to guide the robot through navigation tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Robot NavigationSimilar Papers 제목 키워드 기반
Validating Virtual Reality for Studying Multimodal Human-Robot Interaction in Socially Aware Robot Navigation
Virtual Reality (VR) offers a flexible and controllable platform for studying human-robot interaction. Prior work has explored VR for socially aware robot navigation. However, whether VR captures the multimodal interacti…
Robot NavigationClosed-Loop Open-Vocabulary Mobile Manipulation with GPT-4V
Autonomous robot navigation and manipulation in open environments require reasoning and replanning with closed-loop feedback. In this work, we present COME-robot, the first closed-loop robotic system utilizing the GPT-4V…
Instruction FollowingMultimodal ReasoningRobot NavigationTask PlanningFuture-Oriented Navigation: Dynamic Obstacle Avoidance with One-Shot Energy-Based Multimodal Motion Prediction
This paper proposes an integrated approach for the safe and efficient control of mobile robots in dynamic and uncertain environments. The approach consists of two key steps: one-shot multimodal motion prediction to antic…
Model Predictive ControlMotion Planningmotion predictionNavigate+1ARTOO-DARTU: Studying AR-HRC With AR Obstruction Mitigation During a Warehouse Task
Human-robot collaboration (HRC) often requires robot intentions and internal states to be conveyed to users for task efficiency and safety. Recently, augmented reality (AR) situated analytics provide such real-time robot…
A Bimanual Gesture Interface for ROS-Based Mobile Manipulators Using TinyML and Sensor Fusion
Gesture-based control for mobile manipulators faces persistent challenges in reliability, efficiency, and intuitiveness. This paper presents a dual-hand gesture interface that integrates TinyML, spectral analysis, and se…
Gesture Recognition