Scalable Training of Language Models using JAX pjit and TPUv4

2022-04-13 · Joanna Yoo, Kuba Perlin, Siddhartha Rao Kamalakara, João G. M. Araújo

Modern large language models require distributed training strategies due to their size. The challenges of efficiently and robustly training them are met with rapid developments on both software and hardware frontiers. In this technical report, we explore challenges and design decisions associated with developing a scalable training framework, and present a quantitative analysis of efficiency improvements coming from adopting new software and hardware solutions.

📄 PDF Abstract BibTeX arXiv:2204.06514

Code (0)

등록된 구현이 없습니다.

LAST: Scalable Lattice-Based Speech Modelling in JAX

2023-04-25 · Ke wu, Ehsan Variani, Tom Bagby, Michael Riley

We introduce LAST, a LAttice-based Speech Transducer library in JAX. With an emphasis on flexibility, ease-of-use, and scalability, LAST implements differentiable weighted finite state automaton (WFSA) algorithms needed …

GPU

Simple, Distributed, and Accelerated Probabilistic Programming

2018-11-05 · NeurIPS 2018 12 · Dustin Tran, Matthew Hoffman, Dave Moore, Christopher Suter 외

We describe a simple, low-level approach for embedding probabilistic programming in a deep learning ecosystem. In particular, we distill probabilistic programming down to a single abstraction---the random variable. Our l…

GPU Probabilistic Programming

InfiniteHBD: Building Datacenter-Scale High-Bandwidth Domain for LLM with Optical Circuit Switching Transceivers

2025-02-06 · Chenchen Shou, Guyue Liu, Hao Nie, Huaiyu Meng 외

Scaling Large Language Model (LLM) training relies on multi-dimensional parallelism, where High-Bandwidth Domains (HBDs) are critical for communication-intensive parallelism like Tensor Parallelism (TP) and Expert Parall…

GPU Large Language Model

GSPMD: General and Scalable Parallelization for ML Computation Graphs

2021-05-10 · Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake Hechtman 외

We present GSPMD, an automatic, compiler-based parallelization system for common machine learning computations. It allows users to write programs in the same way as for a single device, then give hints through a few anno…

Playing the Game of 2048

Batch Size Influence on Performance of Graphic and Tensor Processing Units during Training and Inference Phases

2018-12-31 · Yuriy Kochura, Yuri Gordienko, Vlad Taran, Nikita Gordienko 외

The impact of the maximally possible batch size (for the better runtime) on performance of graphic processing units (GPU) and tensor processing units (TPU) during training and inference phases is investigated. The numero…

GPU