Data Compression using Rank-1 Lattices for Parameter Estimation in Machine Learning
The mean squared error and regularized versions of it are standard loss functions in supervised machine learning. However, calculating these losses for large data sets can be computationally demanding. Modifying an approach of J. Dick and M. Feischl [Journal of Complexity 67 (2021)], we present algorithms to reduce extensive data sets to a smaller size using rank-1 lattices. Rank-1 lattices are quasi-Monte Carlo (QMC) point sets that are, if carefully chosen, well-distributed in a multidimensional unit cube. The compression strategy in the preprocessing step assigns every lattice point a pair of weights depending on the original data and responses, representing its relative importance. As a result, the compressed data makes iterative loss calculations in optimization steps much faster. We analyze the errors of our QMC data compression algorithms and the cost of the preprocessing step for functions whose Fourier coefficients decay sufficiently fast so that they lie in certain Wiener algebras or Korobov spaces. In particular, we prove that our approach can lead to arbitrary high convergence rates as long as the functions are sufficiently smooth.
Code (0)
등록된 구현이 없습니다.
Tasks
Data Compressionparameter estimationSimilar Papers 제목 키워드 기반
Spherical Leech Quantization for Visual Tokenization and Generation
Non-parametric quantization has received much attention due to its efficiency on parameters and scalability to a large codebook. In this paper, we present a unified formulation of different non-parametric quantization me…
Image GenerationFeature-based Low-Rank Compression of Large Language Models via Bayesian Optimization
In recent years, large language models (LLMs) have driven advances in natural language processing. Still, their growing scale has increased the computational burden, necessitating a balance between efficiency and perform…
Bayesian OptimizationLow-rank compressionExploring the structure-property relations of thin-walled, 2D extruded lattices using neural networks
This paper investigates the structure-property relations of thin-walled lattices under dynamic longitudinal compression, characterized by their cross-sections and heights. These relations elucidate the interactions of di…
Transfer LearningPC-LoRA: Low-Rank Adaptation for Progressive Model Compression with Knowledge Distillation
Low-rank adaption (LoRA) is a prominent method that adds a small number of learnable parameters to the frozen pre-trained weights for parameter-efficient fine-tuning. Prompted by the question, ``Can we make its represent…
Knowledge DistillationModel Compressionparameter-efficient fine-tuningMARS: Masked Automatic Ranks Selection in Tensor Decompositions
Tensor decomposition methods have proven effective in various applications, including compression and acceleration of neural networks. At the same time, the problem of determining optimal decomposition ranks, which prese…
Tensor Decomposition