An Isometric Stochastic Optimizer

2023-07-24 · Jacob Jackson

The Adam optimizer is the standard choice in deep learning applications. I propose a simple explanation of Adam's success: it makes each parameter's step size independent of the norms of the other parameters. Based on this principle I derive Iso, a new optimizer which makes the norm of a parameter's update invariant to the application of any linear transformation to its inputs and outputs. I develop a variant of Iso called IsoAdam that allows optimal hyperparameters to be transferred from Adam, and demonstrate that IsoAdam obtains a speedup over Adam when training a small Transformer.

📄 PDF Abstract BibTeX arXiv:2307.12979

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Attention 설명 없음

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Muse: Representation Geometry of Muon Beyond Normalized Momentum

2026-07-16 · Da Chang, Qiankun Shi, Lvgang Zhang, Di He 외 arxiv

Muon-style optimizers apply a polar map to matrix momentum, but their updates also depend on the representation of each parameter block before orthogonalization. We study this representation choice as a form of optimizer…

TrasMuon: Trust-Region Adaptive Scaling for Orthogonalized Momentum Optimizers

2026-02-13 · Peng Cheng, Jiucheng Zang, Qingnan Li, Liheng Ma 외 arxiv

Muon-style optimizers leverage Newton-Schulz (NS) iterations to orthogonalize updates, yielding update geometries that often outperform Adam-series methods. However, this orthogonalization discards magnitude information,…

SADAM: Stochastic Adam, A Stochastic Operator for First-Order Gradient-based Optimizer

2022-05-20 · Wei zhang, Yu Bao

In this work, to efficiently help escape the stationary and saddle points, we propose, analyze, and generalize a stochastic strategy performed as an operator for a first-order gradient descent algorithm in order to incre…

Variational Stochastic Gradient Descent for Deep Neural Networks

2024-04-09 · Haotian Chen, Anna Kuzina, Babak Esmaeili, Jakub M Tomczak

Current state-of-the-art optimizers are adaptive gradient-based optimization methods such as Adam. Recently, there has been an increasing interest in formulating gradient-based optimizers in a probabilistic framework for…

image-classification Image Classification Variational Inference

Fine-Tuning Adaptive Stochastic Optimizers: Determining the Optimal Hyperparameter $ε$ via Gradient Magnitude Histogram Analysis

2023-11-20 · Gustavo Silva, Paul Rodriguez

Stochastic optimizers play a crucial role in the successful training of deep neural network models. To achieve optimal model performance, designers must carefully select both model and optimizer hyperparameters. However,…

Language Modeling Language Modelling Machine Translation