On Batch-size Selection for Stochastic Training for Graph Neural Networks
In recent years deep learning has become an important framework for supervised learning. It has been observed that stochastic gradient decent (SGD) method in deep learning networks performs well when the minibatch size is small. In this work, we focus on the importance of batch size selection in Graph Neural Networks (GNN). We provide theoretical analysis based on an estimator that considers the randomness arising from two consecutive layers in GNN, and suggest a guideline for picking the appropriate scale of the batch size. We complement our theoretical results with empirical experiments. We consider the following baseline methods: ClusterGCN, FastGCN, GraphSaint on the following datasets: Ogbn-products, Ogbn-arxiv, Reddit and Pubmed. We demonstrate that in contrast to conventional deep learning models, GNNs benefit from large batch sizes.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Submodular Batch Selection for Training Deep Neural Networks
Mini-batch gradient descent based methods are the de facto algorithms for training neural network architectures today. We introduce a mini-batch selection strategy based on submodular function maximization. Our novel sub…
Combinatorial OptimizationDiversityInformativenessA variable metric mini-batch proximal stochastic recursive gradient algorithm with diagonal Barzilai-Borwein stepsize
Variable metric proximal gradient methods with different metric selections have been widely used in composite optimization. Combining the Barzilai-Borwein (BB) method with a diagonal selection strategy for the metric, th…
Online Batch Selection for Faster Training of Neural Networks
Deep neural networks are commonly trained using stochastic non-convex optimization procedures, which are driven by gradient information estimated on fractions (batches) of the dataset. While it is commonly accepted that …
Stochastic batch size for adaptive regularization in deep network optimization
We propose a first-order stochastic optimization algorithm incorporating adaptive regularization applicable to machine learning problems in deep learning framework. The adaptive regularization is imposed by stochastic pr…
image-classificationImage ClassificationStochastic OptimizationMini-Batch Stochastic ADMMs for Nonconvex Nonsmooth Optimization
With the large rising of complex data, the nonconvex models such as nonconvex loss function and nonconvex regularizer are widely used in machine learning and pattern recognition. In this paper, we propose a class of mini…