Mixture of Efficient Diffusion Experts Through Automatic Interval and Sub-Network Selection

2024-09-23 · Alireza Ganjdanesh, Yan Kang, Yuchen Liu, Richard Zhang, Zhe Lin, Heng Huang

Diffusion probabilistic models can generate high-quality samples. Yet, their sampling process requires numerous denoising steps, making it slow and computationally intensive. We propose to reduce the sampling cost by pruning a pretrained diffusion model into a mixture of efficient experts. First, we study the similarities between pairs of denoising timesteps, observing a natural clustering, even across different datasets. This suggests that rather than having a single model for all time steps, separate models can serve as ``experts'' for their respective time intervals. As such, we separately fine-tune the pretrained model on each interval, with elastic dimensions in depth and width, to obtain experts specialized in their corresponding denoising interval. To optimize the resource usage between experts, we introduce our Expert Routing Agent, which learns to select a set of proper network configurations. By doing so, our method can allocate the computing budget between the experts in an end-to-end manner without requiring manual heuristics. Finally, with a selected configuration, we fine-tune our pruned experts to obtain our mixture of efficient experts. We demonstrate the effectiveness of our method, DiffPruning, across several datasets, LSUN-Church, LSUN-Beds, FFHQ, and ImageNet, on the Latent Diffusion Model architecture.

📄 PDF Abstract BibTeX arXiv:2409.15557

Code (0)

등록된 구현이 없습니다.

Tasks

Denoising

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

Pruning 설명 없음

Latent Diffusion Model Diffusion models applied to latent spaces, which are normally built with (Variational) Autoencoders.

TimeStep Master: Asymmetrical Mixture of Timestep LoRA Experts for Versatile and Efficient Diffusion Models in Vision

2025-03-10 · Shaobin Zhuang, Yiwei Guo, Yanbo Ding, Kunchang Li 외

Diffusion models have driven the advancement of vision generation over the past years. However, it is often difficult to apply these large models in downstream tasks, due to massive fine-tuning cost. Recently, Low-Rank A…

Domain Adaptation

Mixture of Online and Offline Experts for Non-stationary Time Series

2022-02-12 · Zhilin Zhao, Longbing Cao, Yuanyu Wan

We consider a general and realistic scenario involving non-stationary time series, consisting of several offline intervals with different distributions within a fixed offline time horizon, and an online interval that con…

Time Series Transfer Learning

TIDE: Efficient and Lossless MoE Diffusion LLM Inference with I/O-aware Expert Offload

2026-05-19 · Zhiben Chen, Youpeng Zhao, Yang Sui, Jun Wang 외 arxiv

Diffusion Large Language Models (dLLMs) have emerged as a competitive alternative to autoregressive (AR) models, offering better hardware utilization and bidirectional context through parallel block-level decoding. Howev…

Prediction Sets for High-Dimensional Mixture of Experts Models

2022-10-30 · Adel Javanmard, Simeng Shao, Jacob Bien

Large datasets make it possible to build predictive models that can capture heterogenous relationships between the response variable and features. The mixture of high-dimensional linear experts model posits that observat…

Mixture-of-Experts Prediction valid Vocal Bursts Intensity Prediction

Abstracting Robot Manipulation Skills via Mixture-of-Experts Diffusion Policies

2026-01-29 · Ce Hao, Xuanran Zhai, Yaohua Liu, Harold Soh arxiv

Diffusion-based policies have recently shown strong results in robot manipulation, but their extension to multi-task scenarios is hindered by the high cost of scaling model size and demonstrations. We introduce Skill Mix…

Multi-Task Learning Robot Manipulation Transfer Learning