Inter-Layer Scheduling Space Exploration for Multi-model Inference on Heterogeneous Chiplets

2023-12-14 · Mohanad Odema, Hyoukjun Kwon, Mohammad Abdullah Al Faruque

To address increasing compute demand from recent multi-model workloads with heavy models like large language models, we propose to deploy heterogeneous chiplet-based multi-chip module (MCM)-based accelerators. We develop an advanced scheduling framework for heterogeneous MCM accelerators that comprehensively consider complex heterogeneity and inter-chiplet pipelining. Our experiments using our framework on GPT-2 and ResNet-50 models on a 4-chiplet system have shown upto 2.2x and 1.9x increase in throughput and energy efficiency, compared to a monolithic accelerator with an optimized output-stationary dataflow.

📄 PDF Abstract BibTeX arXiv:2312.09401

Code (0)

등록된 구현이 없습니다.

Tasks

Scheduling

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Residual Connection 설명 없음

Weight Decay 설명 없음

Cosine Annealing Cosine Annealing is a type of learning rate schedule that has the effect of starting with a large learning rate that is relatively rapidly decreased to a minimum value before…

Intelligent Resource Scheduling for Co-located Latency-critical Services: A Multi-Model Collaborative Learning Approach

2019-11-26 · Lei Liu

Latency-critical services have been widely deployed in cloud environments. For cost-efficiency, multiple services are usually co-located on a server. Thus, run-time resource scheduling becomes the pivot for QoS control i…

BIG-bench Machine Learning Reinforcement Learning Scheduling

A Codesign of Scheduling and Parallelization for Large Model Training in Heterogeneous Clusters

2024-03-24 · Chunyu Xue, Weihao Cui, Han Zhao, Quan Chen 외

Joint consideration of scheduling and adaptive parallelism offers great opportunities for improving the training efficiency of large models on heterogeneous GPU clusters. However, integrating adaptive parallelism into a …

GPU Scheduling

KAPLA: Pragmatic Representation and Fast Solving of Scalable NN Accelerator Dataflow

2023-06-09 · Zhiyao Li, Mingyu Gao

Dataflow scheduling decisions are of vital importance to neural network (NN) accelerators. Recent scalable NN accelerators support a rich set of advanced dataflow techniques. The problems of comprehensively representing …

Scheduling

Digital Twin-Assisted Efficient Reinforcement Learning for Edge Task Scheduling

2022-08-02 · Xiucheng Wang, Longfei Ma, Haocheng Li, Zhisheng Yin 외

Task scheduling is a critical problem when one user offloads multiple different tasks to the edge server. When a user has multiple tasks to offload and only one task can be transmitted to server at a time, while server p…

Q-Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Agile Reinforcement Learning for Real-Time Task Scheduling in Edge Computing

2025-06-10 · Amin Avan, Akramul Azim, Qusay Mahmoud

Soft real-time applications are becoming increasingly complex, posing significant challenges for scheduling offloaded tasks in edge computing environments while meeting task timing constraints. Moreover, the exponential …

Edge-computing reinforcement-learning Reinforcement Learning Scheduling