ZeRO

2000년 도입 · 논문 9편에서 사용

Zero Redundancy Optimizer (ZeRO) is a sharded data parallel method for distributed training. ZeRODP removes the memory state redundancies across data-parallel processes by partitioning the model states instead of replicating them, and it retains the compute/communication efficiency by retaining the computational granularity and communication volume of DP using a dynamic communication schedule during training.

출처: ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

소개 논문: ZeRO: Memory Optimizations Toward Training Trillion Parameter Models