Spectral Regularization: an Inductive Bias for Sequence Modeling
Various forms of regularization in learning tasks strive for different notions of simplicity. This paper presents a spectral regularization technique, which attaches a unique inductive bias to sequence modeling based on an intuitive concept of simplicity defined in the Chomsky hierarchy. From fundamental connections between Hankel matrices and regular grammars, we propose to use the trace norm of the Hankel matrix, the tightest convex relaxation of its rank, as the spectral regularizer. To cope with the fact that the Hankel matrix is bi-infinite, we propose an unbiased stochastic estimator for its trace norm. Ultimately, we demonstrate experimental results on Tomita grammars, which exhibit the potential benefits of spectral regularization and validate the proposed stochastic estimator.
Code (1)
Tasks
Inductive BiasSimilar Papers 제목 키워드 기반
Aligning Inductive Bias for Data-Efficient Generalization in State Space Models
The remarkable success of modern AI has been closely tied to scaling laws, yet the finite supply of high-quality data makes data efficiency--learning more from less--an increasingly important frontier. A model's inductiv…
Frequency Regularization: Unveiling the Spectral Inductive Bias of Deep Neural Networks
Regularization techniques such as L2 regularization (Weight Decay) and Dropout are fundamental to training deep neural networks, yet their underlying physical mechanisms regarding feature frequency selection remain poorl…
A Cognitive Regularizer for Language Modeling
The uniform information density (UID) hypothesis, which posits that speakers behaving optimally tend to distribute information uniformly across a linguistic signal, has gained traction in psycholinguistics as an explanat…
Inductive BiasLanguage ModelingLanguage ModellingSpectral Diffusion Models on the Sphere
Diffusion models provide a principled framework for generative modeling via stochastic differential equations and time-reversed dynamics. However, extension of spectral diffusion approaches to spherical data raises nontr…
Spectral Regularization for Diffusion Models
Diffusion models are typically trained using pointwise reconstruction objectives that are agnostic to the spectral and multi-scale structure of natural signals. We propose a loss-level spectral regularization framework t…
Audio Generation