Muon-Accelerated Attention Distillation for Real-Time Edge Synthesis via Optimized Latent Diffusion

2025-04-11 · Weiye Chen, Qingen Zhu, Qian Long

Recent advances in visual synthesis have leveraged diffusion models and attention mechanisms to achieve high-fidelity artistic style transfer and photorealistic text-to-image generation. However, real-time deployment on edge devices remains challenging due to computational and memory constraints. We propose Muon-AD, a co-designed framework that integrates the Muon optimizer with attention distillation for real-time edge synthesis. By eliminating gradient conflicts through orthogonal parameter updates and dynamic pruning, Muon-AD achieves 3.2 times faster convergence compared to Stable Diffusion-TensorRT, while maintaining synthesis quality (15% lower FID, 4% higher SSIM). Our framework reduces peak memory to 7GB on Jetson Orin and enables 24FPS real-time generation through mixed-precision quantization and curriculum learning. Extensive experiments on COCO-Stuff and ImageNet-Texture demonstrate Muon-AD's Pareto-optimal efficiency-quality trade-offs. Here, we show a 65% reduction in communication overhead during distributed training and real-time 10s/image generation on edge GPUs. These advancements pave the way for democratizing high-quality visual synthesis in resource-constrained environments.

📄 PDF Abstract BibTeX arXiv:2504.08451

Code (0)

등록된 구현이 없습니다.

Tasks

Image Generation Quantization SSIM Style Transfer Text to Image Generation Text-to-Image Generation

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

Advancing Model Refinement: Muon-Optimized Distillation and Quantization for LLM Deployment

2026-01-14 · Jacob Sander, Brian Jalaian, Venkat R. Dasari arxiv

Large Language Models (LLMs) enable advanced natural language processing but face deployment challenges on resource-constrained edge devices due to high computational, memory, and energy demands. Optimizing these models …

Hyperparameter Optimization Knowledge Distillation

When and Why Grouping Attention Heads Accelerates Muon Optimization

2026-05-09 · Hongtao Zhang, Wenjie Zhou, Wei Chen, Xueqi Cheng arxiv

Muon orthogonalizes matrix updates, but multi-head attention naturally operates at the level of heads. This granularity mismatch raises the question of whether Muon should be applied to the full attention projection, to …

Muon: Training and Trade-offs with Latent Attention and MoE

2025-09-29 · Sushant Mehta, Raj Dandekar, Rajat Dandekar, Sreedath Panat arxiv

We present a comprehensive theoretical and empirical study of the Muon optimizer for training transformers only with a small to medium decoder (30M - 200M parameters), with an emphasis on its mathematical foundations, co…

LiMuon: Light and Fast Muon Optimizer for Large Models

2025-09-18 · Feihu Huang, Yuning Luo, Songcan Chen arxiv

Large models recently are widely applied in machine learning, so efficient training of large models has received widespread attention. More recently, the useful Muon optimizer is specifically designed for matrix-structur…

Stochastic Optimization

FedMuon: Federated Learning with Bias-corrected LMO-based Optimization

2025-09-30 · Yuki Takezawa, Anastasia Koloskova, Xiaowen Jiang, Sebastian U. Stich arxiv

Recently, a new optimization method based on the linear minimization oracle (LMO), called Muon, has been attracting increasing attention since it can train neural networks faster than existing adaptive optimization metho…

Federated Learning