Clustering the Sketch: A Novel Approach to Embedding Table Compression

2022-10-12 · Henry Ling-Hei Tsang, Thomas Dybdahl Ahle

Embedding tables are used by machine learning systems to work with categorical features. In modern Recommendation Systems, these tables can be very large, necessitating the development of new methods for fitting them in memory, even during training. We suggest Clustered Compositional Embeddings (CCE) which combines clustering-based compression like quantization to codebooks with dynamic methods like The Hashing Trick and Compositional Embeddings (Shi et al., 2020). Experimentally CCE achieves the best of both worlds: The high compression rate of codebook-based quantization, but *dynamically* like hashing-based methods, so it can be used during training. Theoretically, we prove that CCE is guaranteed to converge to the optimal codebook and give a tight bound for the number of iterations required.

📄 PDF Abstract BibTeX arXiv:2210.05974

Code (1)

thomasahle/cce 공식 구현 pytorch

Tasks

Click-Through Rate PredictionClusteringQuantization Recommendation Systems

Methods 이 논문이 사용한 방법론

k-Means Clustering k-Means Clustering is a clustering algorithm that divides a training set into $k$ different clusters of examples that are near each other. It works by initializing $k$…

Clustering the Sketch: Dynamic Compression for Embedding Tables

2023-09-21 · NeurIPS 2023 11

CAFE: Towards Compact, Adaptive, and Fast Embedding for Large-scale Recommendation Models

2023-12-06 · Hailin Zhang, Zirui Liu, Boxuan Chen, Yikai Zhao 외

Recently, the growing memory demands of embedding tables in Deep Learning Recommendation Models (DLRMs) pose great challenges for model training and deployment. Existing embedding compression solutions cannot simultaneou…

Feature Importance Philosophy

Scaling Graph Clustering with Distributed Sketches

2020-07-24 · Benjamin W. Priest, Alec Dunton, Geoffrey Sanders

The unsupervised learning of community structure, in particular the partitioning vertices into clusters or communities, is a canonical and well-studied problem in exploratory graph analysis. However, like most graph anal…

Clustering Graph Clustering Stochastic Block Model

Linking Sketch Patches by Learning Synonymous Proximity for Graphic Sketch Representation

2022-11-30 · Sicong Zang, Shikui Tu, Lei Xu

Graphic sketch representations are effective for representing sketches. Existing methods take the patches cropped from sketches as the graph nodes, and construct the edges based on sketch's drawing order or Euclidean dis…

Denoising

Mean Nyström Embeddings for Adaptive Compressive Learning

2021-10-21 · Antoine Chatalic, Luigi Carratino, Ernesto de Vito, Lorenzo Rosasco

Compressive learning is an approach to efficient large scale learning based on sketching an entire dataset to a single mean embedding (the sketch), i.e. a vector of generalized moments. The learning task is then approxim…