EinSort: Sorting is All We Need for Tensorizing LLM
Tensor networks provide efficient representations for compressing large neural networks. By carefully designing shapes and topologies, they can significantly reduce memory and computational costs. However, identifying implicit low-rank structures in large foundation models remains challenging due to their enormous scale and un-structured weight distributions. We propose an adaptive tensorization method that discovers inherent low-rank structure in a target tensor by index ordering. Experiments on weight and KV-cache compression demonstrate improved reconstruction quality compared to baselines.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Tensorizing GAN with High-Order Pooling for Alzheimer's Disease Assessment
It is of great significance to apply deep learning for the early diagnosis of Alzheimer's Disease (AD). In this work, a novel tensorizing GAN with high-order pooling is proposed to assess Mild Cognitive Impairment (MCI) …
Vocal Bursts Intensity PredictionHigh-dimensional density estimation with tensorizing flow
We propose the tensorizing flow method for estimating high-dimensional probability density functions from the observed data. The method is based on tensor-train and flow-based generative modeling. Our method first effici…
Density EstimationVocal Bursts Intensity PredictionSTN: Scalable Tensorizing Networks via Structure-Aware Training and Adaptive Compression
Deep neural networks (DNNs) have delivered a remarkable performance in many tasks of computer vision. However, over-parameterized representations of popular architectures dramatically increase their computational complex…
Tensor DecompositionGeneralized Neural Sorting Networks with Error-Free Differentiable Swap Functions
Sorting is a fundamental operation of all computer systems, having been a long-standing significant research topic. Beyond the problem formulation of traditional sorting algorithms, we consider sorting problems for more …
A simulation driven optimization algorithm for scheduling sorting center operations
Parcel sorting operations in logistics enterprises aim to achieve a high throughput of parcels through sorting centers. These sorting centers are composed of large circular conveyor belts on which incoming parcels are pl…
Scheduling