LEMMo-Plan: LLM-Enhanced Learning from Multi-Modal Demonstration for Planning Sequential Contact-Rich Manipulation Tasks

2024-09-18 · KeJia Chen, Zheng Shen, Yue Zhang, Lingyun Chen, Fan Wu, Zhenshan Bing, Sami Haddadin, Alois Knoll

Large Language Models (LLMs) have gained popularity in task planning for long-horizon manipulation tasks. To enhance the validity of LLM-generated plans, visual demonstrations and online videos have been widely employed to guide the planning process. However, for manipulation tasks involving subtle movements but rich contact interactions, visual perception alone may be insufficient for the LLM to fully interpret the demonstration. Additionally, visual data provides limited information on force-related parameters and conditions, which are crucial for effective execution on real robots. In this paper, we introduce an in-context learning framework that incorporates tactile and force-torque information from human demonstrations to enhance LLMs' ability to generate plans for new task scenarios. We propose a bootstrapped reasoning pipeline that sequentially integrates each modality into a comprehensive task plan. This task plan is then used as a reference for planning in new task configurations. Real-world experiments on two different sequential manipulation tasks demonstrate the effectiveness of our framework in improving LLMs' understanding of multi-modal demonstrations and enhancing the overall planning performance.

📄 PDF Abstract BibTeX arXiv:2409.11863

Code (0)

등록된 구현이 없습니다.

Tasks

Contact-rich ManipulationIn-Context LearningTask Planning

RLEMMO: Evolutionary Multimodal Optimization Assisted By Deep Reinforcement Learning

2024-04-12 · Hongqiao Lian, Zeyuan Ma, Hongshu Guo, Ting Huang 외

Solving multimodal optimization problems (MMOP) requires finding all optimal solutions, which is challenging in limited function evaluations. Although existing works strike the balance of exploration and exploitation thr…

Deep Reinforcement Learning Diversity reinforcement-learning Reinforcement Learning

Real-time Vehicle-to-Vehicle Communication Based Network Cooperative Control System through Distributed Database and Multimodal Perception: Demonstrated in Crossroads

2024-10-23 · Xinwen Zhu, Zihao Li, YuXuan Jiang, Jiazhen Xu 외

The autonomous driving industry is rapidly advancing, with Vehicle-to-Vehicle (V2V) communication systems highlighting as a key component of enhanced road safety and traffic efficiency. This paper introduces a novel Real…

Autonomous Driving Autonomous Vehicles Collision Avoidance

SPADE: Sketch-guided Path Planning Augmented with Diffusion Experts

2026-06-02 · Charbel Abi Hana, Tatiana Ghantous, Mikael Khalil, Anthony Rizk arxiv

Path planning is essential for Autonomous Mobile Robots (AMRs). Conventional methods for incorporating human preferences into planning typically rely on either complex reward engineering or hardware-intensive solutions. …

MAF: Multimodal Adaptive Few-shot Prompting for Sentiment Analysis with MLLMs

2026-06-14 · Hangling Xie arxiv

Multimodal large language models (MLLMs) have demonstrated remarkable capabilities in understanding complex multimodal content. However, their performance in sentiment analysis exhibits acute sensitivity to prompt design…

Speaker Identification Sentiment Analysis

Retrieval-augmented in-context learning for multimodal large language models in disease classification

2025-05-04 · Zaifu Zhan, Shuang Zhou, Xiaoshan Zhou, Yongkang Xiao 외

Objectives: We aim to dynamically retrieve informative demonstrations, enhancing in-context learning in multimodal large language models (MLLMs) for disease classification. Methods: We propose a Retrieval-Augmented In-Co…

In-Context Learning RAG Retrieval Retrieval-augmented Generation