GShard

2000년 도입 · 논문 6편에서 사용

GShard is a intra-layer parallel distributed method. It consists of set of simple APIs for annotations, and a compiler extension in XLA for automatic parallelization.

출처: GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

소개 논문: GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding