On Batch Orthogonalization Layers
Batch normalization has become ubiquitous in many state-of-the-art nets. It accelerates training and yields good performance results. However, there are various other alternatives to normalization, e.g. orthonormalization. The objective of this paper is to explore the possible alternatives to channel normalization with orthonormalization layers. The performance of the algorithms are compared together with BN with prescribed performance measures.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Spectral Gradient Orthogonalization Improves Differentially Private Training at Scale
Differentially private training adds isotropic Gaussian noise to clipped gradients, corrupting every singular direction equally. In vision models, where spatial correlation concentrates gradient energy into a low-rank su…
SpecNet2: Orthogonalization-free spectral embedding by neural networks
Spectral methods which represent data points by eigenvectors of kernel matrices or graph Laplacian matrices have been a primary tool in unsupervised data analysis. In many application scenarios, parametrizing the spectra…
Computational EfficiencyPro-KLShampoo: Projected KL-Shampoo with Whitening Recovered by Orthogonalization
Optimizers that exploit the matrix structure of gradients are central to modern LLM pre-training, with two distinct frontiers: explicit Kronecker-factored preconditioning -- most recently KL-Shampoo, which estimates the …
Dion3: Full-Stack Orthogonal Updates
The Muon optimizer incurs a significant overhead cost due to its cubic-time Newton-Schulz orthogonalization step. When weights are sharded, communication overhead compounds this computational cost, eroding the benefits o…
A Unified Batch Selection Policy for Active Metric Learning
Active metric learning is the problem of incrementally selecting high-utility batches of training data (typically, ordered triplets) to annotate, in order to progressively improve a learned model of a metric over some in…
Active LearningDiversityInformativenessMetric Learning+1