Iterative Averaging in the Quest for Best Test Error

2020-03-02 · Diego Granziol, Xingchen Wan, Samuel Albanie, Stephen Roberts

We analyse and explain the increased generalisation performance of iterate averaging using a Gaussian process perturbation model between the true and batch risk surface on the high dimensional quadratic. We derive three phenomena \latestEdits{from our theoretical results:} (1) The importance of combining iterate averaging (IA) with large learning rates and regularisation for improved regularisation. (2) Justification for less frequent averaging. (3) That we expect adaptive gradient methods to work equally well, or better, with iterate averaging than their non-adaptive counterparts. Inspired by these results\latestEdits{, together with} empirical investigations of the importance of appropriate regularisation for the solution diversity of the iterates, we propose two adaptive algorithms with iterate averaging. These give significantly better results compared to stochastic gradient descent (SGD), require less tuning and do not require early stopping or validation set monitoring. We showcase the efficacy of our approach on the CIFAR-10/100, ImageNet and Penn Treebank datasets on a variety of modern and classical network architectures.

📄 PDF Abstract BibTeX arXiv:2003.01247

Code (0)

등록된 구현이 없습니다.

Tasks

DiversityImage Classification

Methods 이 논문이 사용한 방법론

Weight Decay 설명 없음

Adam 설명 없음

SGD Stochastic Gradient Descent is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using…

Revisiting SGD with Increasingly Weighted Averaging: Optimization and Generalization Perspectives

2020-03-09 · Zhishuai Guo, Yan Yan, Tianbao Yang

Stochastic gradient descent (SGD) has been widely studied in the literature from different angles, and is commonly employed for solving many big data machine learning problems. However, the averaging technique, which com…

Distributed linear regression by averaging

2018-09-30 · Edgar Dobriban, Yue Sheng

Distributed statistical learning problems arise commonly when dealing with large datasets. In this setup, datasets are partitioned over machines, which compute locally, and communicate short messages. Communication is of…

regression

Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimization

2018-06-03 · NeurIPS 2018 12 · Hoi-To Wai, Zhuoran Yang, Zhaoran Wang, Mingyi Hong

Despite the success of single-agent reinforcement learning, multi-agent reinforcement learning (MARL) remains challenging due to complex interactions between agents. Motivated by decentralized applications such as sensor…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Sparse Mutual Information Graph Averaging for Improving Random Indexing Embeddings

2026-08-06 · Sriram Loganathan, Gokul Anand, Aung Bo Bo, Yourui Shao 외 arxiv

Sparse word embedding pipelines can avoid dense co-occurrence matrix materialization, dense factorization, and gradient training while still relying on sparse global corpus statistics. This paper studies Random Indexing …

Adaptive-WAM: Quality-Guided Early-Exit Planning from Intermediate Video-Diffusion Features

2026-08-06 · Sining Ang, Yuguang Yang, Yan Wang arxiv

Large video diffusion models provide rich spatiotemporal priors for autonomous driving, but existing world-action models often inherit the cost of iterative future-video generation even though deployment only requires an…

Autonomous Driving Video Generation Video Denoising