Mesh-TensorFlow

2000년 도입 · 논문 2편에서 사용

Mesh-TensorFlow is a language for specifying a general class of distributed tensor computations. Where data-parallelism can be viewed as splitting tensors and operations along the "batch" dimension, in Mesh-TensorFlow, the user can specify any tensor dimensions to be split across any dimensions of a multi-dimensional mesh of processors. A MeshTensorFlow graph compiles into a SPMD program consisting of parallel operations coupled with collective communication primitives such as Allreduce.

출처: Mesh-TensorFlow: Deep Learning for Supercomputers

소개 논문: Mesh-TensorFlow: Deep Learning for Supercomputers