Modality-Composable Diffusion Policy via Inference-Time Distribution-level Composition

2025-03-16 · Jiahang Cao, Qiang Zhang, Hanzhong Guo, Jiaxu Wang, Hao Cheng, Renjing Xu

Diffusion Policy (DP) has attracted significant attention as an effective method for policy representation due to its capacity to model multi-distribution dynamics. However, current DPs are often based on a single visual modality (e.g., RGB or point cloud), limiting their accuracy and generalization potential. Although training a generalized DP capable of handling heterogeneous multimodal data would enhance performance, it entails substantial computational and data-related costs. To address these challenges, we propose a novel policy composition method: by leveraging multiple pre-trained DPs based on individual visual modalities, we can combine their distributional scores to form a more expressive Modality-Composable Diffusion Policy (MCDP), without the need for additional training. Through extensive empirical experiments on the RoboTwin dataset, we demonstrate the potential of MCDP to improve both adaptability and performance. This exploration aims to provide valuable insights into the flexible composition of existing DPs, facilitating the development of generalizable cross-modality, cross-domain, and even cross-embodiment policies. Our code is open-sourced at https://github.com/AndyCao1125/MCDP.

📄 PDF Abstract BibTeX arXiv:2503.12466

Code (1)

andycao1125/mcdp 공식 구현 pytorch

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

Training-Free Imitation Learning with Closed-Form Diffusion Policies

2026-05-31 · Raghav Mishra, Ian R. Manchester arxiv

While diffusion-based policies have impressive performance and expressivity, their long offline training slows down the data collection and policy deployment loop. We introduce Closed-Form Diffusion Policies, a class of …

Any-to-Any Generation via Composable Diffusion

2023-05-19 · NeurIPS 2023 11 · Zineng Tang, ZiYi Yang, Chenguang Zhu, Michael Zeng 외

We present Composable Diffusion (CoDi), a novel generative model capable of generating any combination of output modalities, such as language, image, video, or audio, from any combination of input modalities. Unlike exis…

Audio Generation

Enhanced DACER Algorithm with High Diffusion Efficiency

2025-05-29 · Yinuo Wang, Mining Tan, Wenjun Zou, Haotian Lin 외

Due to their expressive capacity, diffusion models have shown great promise in offline RL and imitation learning. Diffusion Actor-Critic with Entropy Regulator (DACER) extended this capability to online RL by using the r…

Denoising Imitation Learning MuJoCo Offline RL+1

Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Model

2025-10-31 · John Won, Kyungmin Lee, Huiwon Jang, Dongyoung Kim 외 arxiv

Augmenting vision-language-action models (VLAs) with world models is promising for robotic policy learning but faces challenges in jointly predicting states and actions due to the modality gap. To address this, we propos…

Transfer Learning

CoLA-Flow Policy: Temporally Coherent Imitation Learning via Continuous Latent Action Flow Matching for Robotic Manipulation

2026-01-30 · Wu Songwei, Jiang Zhiduo, Sun Wandong, Xie Guanghu 외 arxiv

Learning long-horizon robotic manipulation requires jointly achieving expressive behavior modeling, real-time inference, and stable execution, which remains challenging for existing generative policies. Diffusion-based a…