FusionStitching: Boosting Memory Intensive Computations for Deep Learning Workloads
We show in this work that memory intensive computations can result in severe performance problems due to off-chip memory access and CPU-GPU context switch overheads in a wide range of deep learning models. For this problem, current just-in-time (JIT) kernel fusion and code generation techniques have limitations, such as rough fusion plan exploration strategies and limited code generation ability. We propose FusionStitching, a deep learning compiler capable of fusing memory intensive operators, with varied data dependencies and non-homogeneous parallelism, into large GPU kernels to reduce global memory access and context switch overhead automatically. FusionStitching widens the range of operation combinations that fusion can target beyond previous JIT works by introducing data reuse of intermediate values. It explores large fusion spaces to decide optimal fusion plans with considerations of memory access costs, kernel calls and resource usage constraints. FusionStitching tunes the optimal stitching scheme with a domain-specific cost model efficiently. Experimental results show that FusionStitching can reach up to 2.21x speedup compared to state-of-the-art, with 1.45x on average. Besides these experimental results, we integrated our approach into a compiler product and deployed it onto a production cluster for AI workloads with thousands of GPUs. The system has been in operation for more than 4 months and saves 7,000 GPU hours on average for approximately 30,000 tasks per month.
Code (0)
등록된 구현이 없습니다.
Tasks
Code GenerationCPUDeep LearningGPUSimilar Papers 제목 키워드 기반
TENSILE: A Tensor granularity dynamic GPU memory scheduling method toward multiple dynamic workloads system
Recently, deep learning has been an area of intense research. However, as a kind of computing-intensive task, deep learning highly relies on the scale of GPU memory, which is usually prohibitive and scarce. Although some…
Deep LearningGPUManagementSchedulingBrainSlug: Transparent Acceleration of Deep Learning Through Depth-First Parallelism
Neural network frameworks such as PyTorch and TensorFlow are the workhorses of numerous machine learning applications ranging from object recognition to machine translation. While these frameworks are versatile and strai…
Deep LearningMachine TranslationObject RecognitionTranslationPIM-Opt: Demystifying Distributed Optimization Algorithms on a Real-World Processing-In-Memory System
Modern Machine Learning (ML) training on large-scale datasets is a very time-consuming workload. It relies on the optimization algorithm Stochastic Gradient Descent (SGD) due to its effectiveness, simplicity, and general…
CPUDistributed OptimizationGPUPerun: Secure Multi-Stakeholder Machine Learning Framework with GPU Support
Confidential multi-stakeholder machine learning (ML) allows multiple parties to perform collaborative data analytics while not revealing their intellectual property, such as ML source code, model, or datasets. State-of-t…
BIG-bench Machine LearningGPUSparseTem: Boosting the Efficiency of CNN-Based Video Encoders by Exploiting Temporal Continuity
Deep learning models have become pivotal in the field of video processing and is increasingly critical in practical applications such as autonomous driving and object detection. Although Vision Transformers (ViTs) have d…
Autonomous Drivingobject-detectionObject DetectionScheduling