AutoSync

2000년 도입 · 논문 1편에서 사용

AutoSync is a pipeline for automatically optimizing synchronization strategies, given model structures and resource specifications, in data-parallel distributed machine learning. By factorizing the synchronization strategy with respect to each trainable building block of a DL model, we can construct a valid and large strategy space spanned by multiple factors. AutoSync efficiently navigates the space and locates the optimal strategy. AutoSync leverages domain knowledge about synchronization systems to reduce the search space, and is equipped with a domain adaptive simulator, which combines principled communication modeling and data-driven ML models, to estimate the runtime of strategy proposals without launching real distributed execution.

출처: AutoSync: Learning to Synchronize for Data-Parallel Distributed Deep Learning

소개 논문: AutoSync: Learning to Synchronize for Data-Parallel Distributed Deep Learning