Improving Generalization and Convergence by Enhancing Implicit Regularization

2024-05-31 · Mingze Wang, Jinbo Wang, Haotian He, Zilin Wang, Guanhua Huang, Feiyu Xiong, Zhiyu Li, Weinan E, Lei Wu

In this work, we propose an Implicit Regularization Enhancement (IRE) framework to accelerate the discovery of flat solutions in deep learning, thereby improving generalization and convergence. Specifically, IRE decouples the dynamics of flat and sharp directions, which boosts the sharpness reduction along flat directions while maintaining the training stability in sharp directions. We show that IRE can be practically incorporated with {\em generic base optimizers} without introducing significant computational overload. Experiments show that IRE consistently improves the generalization performance for image classification tasks across a variety of benchmark datasets (CIFAR-10/100, ImageNet) and models (ResNets and ViTs). Surprisingly, IRE also achieves a $2\times$ {\em speed-up} compared to AdamW in the pre-training of Llama models (of sizes ranging from 60M to 229M) on datasets including Wikitext-103, Minipile, and Openwebtext. Moreover, we provide theoretical guarantees, showing that IRE can substantially accelerate the convergence towards flat minima in Sharpness-aware Minimization (SAM).

📄 PDF Abstract BibTeX arXiv:2405.20763

Code (1)

wmz9/ire-algorithm-framework 공식 구현 pytorch

Tasks

image-classification Image Classification

Methods 이 논문이 사용한 방법론

Sharpness-Aware Minimization Sharpness-Aware Minimization, or SAM, is a procedure that improves model generalization by simultaneously minimizing loss value and loss sharpness. SAM functions by…

BASE 설명 없음

AdamW AdamW is a stochastic optimization method that modifies the typical implementation of weight decay in Adam, by decoupling [weight…

LLaMA LLaMA is a collection of foundation language models ranging from 7B to 65B parameters. It is based on the transformer architecture with various improvements that were…

Representation Convergence: Mutual Distillation is Secretly a Form of Regularization

2025-01-05 · Zhengpeng Xie, Jiahang Cao, Qiang Zhang, Jianxiong Zhang 외

In this paper, we argue that mutual distillation between reinforcement learning policies serves as an implicit regularization, preventing them from overfitting to irrelevant features. We highlight two key contributions: …

Deep Reinforcement Learning Form reinforcement-learning Reinforcement Learning+3

A Unified Approach to Controlling Implicit Regularization via Mirror Descent

2023-06-24 · Haoyuan Sun, Khashayar Gatmiry, Kwangjun Ahn, Navid Azizan

Inspired by the remarkable success of large neural networks, there has been significant interest in understanding the generalization performance of over-parameterized models. Substantial efforts have been invested in cha…

Classification regression

Towards Unraveling and Improving Generalization in World Models

2024-12-31 · Qiaoyi Fang, Weiyu Du, Hang Wang, Junshan Zhang

World models have recently emerged as a promising approach to reinforcement learning (RL), achieving state-of-the-art performance across a wide range of visual control tasks. This work aims to obtain a deep understanding…

Reinforcement Learning (RL)

Conflicting Biases at the Edge of Stability: Norm versus Sharpness Regularization

2025-05-27 · Vit Fojtik, Maria Matveev, Hung-Hsu Chou, Gitta Kutyniok 외

A widely believed explanation for the remarkable generalization capacities of overparameterized neural networks is that the optimization algorithms used for training induce an implicit bias towards benign solutions. To g…

Implicit Gradient Alignment in Distributed and Federated Learning

2021-06-25 · Yatin Dandi, Luis Barba, Martin Jaggi

A major obstacle to achieving global convergence in distributed and federated learning is the misalignment of gradients across clients, or mini-batches due to heterogeneity and stochasticity of the distributed data. In t…

Federated Learning