Optimizing Deep Learning Recommender Systems' Training On CPU Cluster Architectures
During the last two years, the goal of many researchers has been to squeeze the last bit of performance out of HPC system for AI tasks. Often this discussion is held in the context of how fast ResNet50 can be trained. Unfortunately, ResNet50 is no longer a representative workload in 2020. Thus, we focus on Recommender Systems which account for most of the AI cycles in cloud computing centers. More specifically, we focus on Facebook's DLRM benchmark. By enabling it to run on latest CPU hardware and software tailored for HPC, we are able to achieve more than two-orders of magnitude improvement in performance (110x) on a single socket compared to the reference CPU implementation, and high scaling efficiency up to 64 sockets, while fitting ultra-large datasets. This paper discusses the optimization techniques for the various operators in DLRM and which component of the systems are stressed by these different operators. The presented techniques are applicable to a broader set of DL workloads that pose the same scaling challenges/characteristics as DLRM.
Code (2)
Tasks
Cloud ComputingCPUDeep LearningRecommendation SystemsSimilar Papers 제목 키워드 기반
NASRec: Weight Sharing Neural Architecture Search for Recommender Systems
The rise of deep neural networks offers new opportunities in optimizing recommender systems. However, optimizing recommender systems using deep neural networks requires delicate architecture fabrication. We propose NASRe…
Click-Through Rate PredictionNeural Architecture SearchRecommendation SystemsRethinking Recommender Systems: Cluster-based Algorithm Selection
Cluster-based algorithm selection deals with selecting recommendation algorithms on clusters of users to obtain performance gains. No studies have been attempted for many combinations of clustering approaches and recomme…
ClusteringRecommendation SystemsGreen Recommender Systems: Optimizing Dataset Size for Energy-Efficient Algorithm Performance
As recommender systems become increasingly prevalent, the environmental impact and energy efficiency of training large-scale models have come under scrutiny. This paper investigates the potential for energy-efficient alg…
Recommendation SystemsGraphHash: Graph Clustering Enables Parameter Efficiency in Recommender Systems
Deep recommender systems rely heavily on large embedding tables to handle high-cardinality categorical features such as user/item identifiers, and face significant memory constraints at scale. To tackle this challenge, h…
Click-Through Rate PredictionClusteringGraph ClusteringRecommendation SystemsReview of Clustering-Based Recommender Systems
Recommender systems are one of the most applied methods in machine learning and find applications in many areas, ranging from economics to the Internet of things. This article provides a general overview of modern approa…
ClusteringDiversityRecommendation Systems