Spectral Compact Training: Pre-Training Large Language Models via Permanent Truncated SVD and Stiefel QR Retraction

2026-04-01 · Björn Roman Kohlberger arxiv

The memory wall remains the primary bottleneck for training large language models on consumer hardware. We introduce Spectral Compact Training (SCT), a method that replaces dense weight matrices with permanent truncated SVD factors W = U diag(s) V^T, where the full dense matrix is never materialized during training or inference. Gradients flow through the compact spectral factors via standard backpropagation, and U, V are retracted to the Stiefel manifold via QR decomposition after each optimizer step. SCT achieves up to 199x memory reduction per MLP layer at rank 32, enabling full training steps of 70B-parameter architectures on a Steam Deck handheld (7.2 GB peak memory vs. 1,245 GB for dense FP32 training with Adam). Rank-sweep experiments on SmolLM2-1.7B (ranks 32-256, 2000 steps, NVIDIA A100) show that all tested ranks converge to the same loss floor (~4.2-4.5), identifying the learning rate schedule -- not MLP rank -- as the primary bottleneck. Rank 128 emerges as the efficiency sweet spot at 11.7x MLP compression with the lowest perplexity. GPU memory drops 46% at rank 32 while training throughput doubles.

📄 PDF Abstract BibTeX arXiv:2604.00733

Code (0)

등록된 구현이 없습니다.

Spectral Signatures of Large Language Models

2026-07-03 · Zhuoying Zhang, Ishan V. Prasad, Yuanzhe Hu, Zihang Liu 외 arxiv

The rapidly growing repository of publicly available large language models (LLMs) presents significant challenges for systematic management and quantification at scale, such as model lineage tracing, licensing, and evalu…

Diversity in Spectral Learning for Natural Language Parsing

2015-05-31 · EMNLP 2015 9 · Shashi Narayan, Shay B. Cohen

We describe an approach to create a diverse set of predictions with spectral learning of latent-variable PCFGs (L-PCFGs). Our approach works by creating multiple spectral models where noise is added to the underlying fea…

Diversity

Spectral Tensor Train Parameterization of Deep Learning Layers

2021-03-07 · Anton Obukhov, Maxim Rakhuba, Alexander Liniger, Zhiwu Huang 외

We study low-rank parameterizations of weight matrices with embedded spectral properties in the Deep Learning context. The low-rank property leads to parameter efficiency and permits taking computational shortcuts when c…

Deep Learning image-classification Image Classification Image Generation+1

S2MAE: A Spatial-Spectral Pretraining Foundation Model for Spectral Remote Sensing Data

2024-01-01 · CVPR 2024 1 · Xuyang Li, Danfeng Hong, Jocelyn Chanussot

In the expansive domain of computer vision a myriad of pre-trained models are at our disposal. However most of these models are designed for natural RGB images and prove inadequate for spectral remote sensing (RS) im…

Spectral Rewiring for Exploration, Purification, and Model Merging

2026-07-03 · Zhilong Zhang, Hongli Yu, Huan-ang Gao, Hanlin Wu 외 arxiv

Reinforcement learning has become a standard post-training recipe for large language models, but dense full-parameter updates create two deployment-relevant bottlenecks: suppressed reasoning performance, often reflected …

Reinforcement Learning Mathematical Reasoning Domain Generalization Instruction Following