홈 › Papers

Static Batching of Irregular Workloads on GPUs: Framework and Application to Efficient MoE Model Inference

2025-01-27 · Yinghan Li, Yifei Li, Jiejing Zhang, Bujiao Chen, Xiaotong Chen, Lian Duan, Yejun Jin, Zheng Li, Xuanyu Liu, Haoyu Wang, Wente Wang, Yajie Wang, Jiacheng Yang, Peiyang Zhang, Laiwen Zheng, Wenyuan Yu

It has long been a problem to arrange and execute irregular workloads on massively parallel devices. We propose a general framework for statically batching irregular workloads into a single kernel with a runtime task mapping mechanism on GPUs. We further apply this framework to Mixture-of-Experts (MoE) model inference and implement an optimized and efficient CUDA kernel. Our MoE kernel achieves up to 91% of the peak Tensor Core throughput on NVIDIA H800 GPU and 95% on NVIDIA H20 GPU.

📄 PDF Abstract BibTeX arXiv:2501.16103

Code (0)

등록된 구현이 없습니다.

Tasks

GPUMixture-of-Experts

Methods 이 논문이 사용한 방법론

MoE 설명 없음

FlashMem: Supporting Modern DNN Workloads on Mobile with GPU Memory Hierarchy Optimizations

2026-02-17 · Zhihao Shu, Md Musfiqur Rahman Sanim, Hangyu Zheng, Kunxiong Zhu 외 arxiv

The increasing size and complexity of modern deep neural networks (DNNs) pose significant challenges for on-device inference on mobile GPUs, with limited memory and computational resources. Existing DNN acceleration fram…

HYDRA: A Heterogeneous Chiplet DSE Framework for Serving Dynamic Hybrid LLM Workloads

2026-08-19 · Jiahao Lin, Alish Kanani, Sangwan Lee, Jaehyun Park 외 arxiv

Hybrid Transformer-Mamba large language models (LLMs) enhance long-context efficiency, but their heterogeneous computation and communication patterns complicate efficient hardware acceleration. Chiplet-based architecture…

MLProxy: SLA-Aware Reverse Proxy for Machine Learning Inference Serving on Serverless Computing Platforms

2022-02-23 · Nima Mahmoudi, Hamzeh Khazaei

Serving machine learning inference workloads on the cloud is still a challenging task on the production level. Optimal configuration of the inference workload to meet SLA requirements while optimizing the infrastructure …

BIG-bench Machine Learning Management

Understanding and Optimizing Multi-Stage AI Inference Pipelines

2025-04-14 · Abhimanyu Rajeshkumar Bambhaniya, Hanjiang Wu, Suvinay Subramanian, Sudarshan Srinivasan 외

The rapid evolution of Large Language Models (LLMs) has driven the need for increasingly sophisticated inference pipelines and hardware platforms. Modern LLM serving extends beyond traditional prefill-decode workflows, i…

CPU Navigate RAG Retrieval+1

Improved Batching Strategy For Irregular Time-Series ODE

2022-07-12 · Ting Fung Lam, Yony Bresler, Ahmed Khorshid, Nathan Perlmutter

Irregular time series data are prevalent in the real world and are challenging to model with a simple recurrent neural network (RNN). Hence, a model that combines the use of ordinary differential equations (ODE) and RNN …

Irregular Time Series Time Series Time Series Analysis