Amazon SageMaker Model Parallelism: A General and Flexible Framework for Large Model Training
With deep learning models rapidly growing in size, systems-level solutions for large-model training are required. We present Amazon SageMaker model parallelism, a software library that integrates with PyTorch, and enables easy training of large models using model parallelism and other memory-saving features. In contrast to existing solutions, the implementation of the SageMaker library is much more generic and flexible, in that it can automatically partition and run pipeline parallelism over arbitrary model architectures with minimal code change, and also offers a general and extensible framework for tensor parallelism, which supports a wider range of use cases, and is modular enough to be easily applied to new training scripts. The library also preserves the native PyTorch user experience to a much larger degree, supporting module re-use and dynamic graphs, while giving the user full control over the details of the training step. We evaluate performance over GPT-3, RoBERTa, BERT, and neural collaborative filtering, and demonstrate competitive performance over existing solutions.
Code (0)
등록된 구현이 없습니다.
Tasks
Collaborative FilteringmodelMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Amazon SageMaker Clarify: Machine Learning Bias Detection and Explainability in the Cloud
Understanding the predictions made by machine learning (ML) models and their potential biases remains a challenging and labor-intensive task that depends on the application, the dataset, and the specific model. We presen…
Bias DetectionBIG-bench Machine LearningFairnessFeature ImportanceAmazon SageMaker Automatic Model Tuning: Scalable Gradient-Free Optimization
Tuning complex machine learning systems is challenging. Machine learning typically requires to set hyperparameters, be it regularization, architecture, or optimization parameters, whose tuning is critical to achieve good…
Bayesian OptimizationBIG-bench Machine LearningAmazon SageMaker Model Monitor: A System for Real-Time Insights into Deployed Machine Learning Models
With the increasing adoption of machine learning (ML) models and systems in high-stakes settings across different industries, guaranteeing a model's performance after deployment has become crucial. Monitoring models in p…
BIG-bench Machine LearningAmazon SageMaker Autopilot: a white box AutoML solution at scale
AutoML systems provide a black-box solution to machine learning problems by selecting the right way of processing features, choosing an algorithm and tuning the hyperparameters of the entire pipeline. Although these syst…
AutoMLMeta-LearningSolving cold start in news recommendations: a RippleNet-based system for large scale media outlet
We present a scalable recommender system implementation based on RippleNet, tailored for the media domain with a production deployment in Onet.pl, one of Poland's largest online media platforms. Our solution addresses th…