MONGOOSE: A Learnable LSH Framework for Efficient Neural Network Training
Recent advances by practitioners in the deep learning community have breathed new life into Locality Sensitive Hashing (LSH), using it to reduce memory and time bottlenecks in neural network (NN) training. However, while LSH has sub-linear guarantees for approximate near-neighbor search in theory, it is known to have inefficient query time in practice due to its use of random hash functions. Moreover, when model parameters are changing, LSH suffers from update overhead. This work is motivated by an observation that model parameters evolve slowly, such that the changes do not always require an LSH update to maintain performance. This phenomenon points to the potential for a reduction in update time and allows for a modified learnable version of data-dependent LSH to improve query time at a low cost. We use the above insights to build MONGOOSE, an end-to-end LSH framework for efficient NN training. In particular, MONGOOSE is equipped with a scheduling algorithm to adaptively perform LSH updates with provable guarantees and learnable hash functions to improve query efficiency. Empirically, we validate MONGOOSE on large-scale deep learning models for recommendation systems and language modeling. We find that it achieves up to $8\%$ better accuracy compared to previous LSH approaches, with $3 \times$ speed-up.
Code (0)
등록된 구현이 없습니다.
Tasks
Efficient Neural NetworkLanguage ModelingLanguage ModellingRecommendation SystemsSchedulingSimilar Papers 제목 키워드 기반
MONGOOSE: Path-wise Smooth Bayesian Optimisation via Meta-learning
In Bayesian optimisation, we often seek to minimise the black-box objective functions that arise in real-world physical systems. A primary contributor to the cost of evaluating such black-box objective functions is often…
Bayesian OptimisationMeta-LearningA Quantum Tunneling and Bio-Phototactic Driven Enhanced Dwarf Mongoose Optimizer for UAV Trajectory Planning and Engineering Problem
With the widespread adoption of unmanned aerial vehicles (UAV), effective path planning has become increasingly important. Although traditional search methods have been extensively applied, metaheuristic algorithms have …
Trajectory PlanningUnlearnable 3D Point Clouds: Class-wise Transformation Is All You Need
Traditional unlearnable strategies have been proposed to prevent unauthorized users from training on the 2D image data. With more 3D point cloud data containing sensitivity information, unauthorized usage of this new typ…
AllUnSeg: One Universal Unlearnable Example Generator is Enough against All Image Segmentation
Image segmentation is a crucial vision task that groups pixels within an image into semantically meaningful segments, which is pivotal in obtaining a fine-grained understanding of real-world scenes. However, an increasin…
AllBilevel OptimizationImage SegmentationInteractive Segmentation+2Unlearnable Clusters: Towards Label-agnostic Unlearnable Examples
There is a growing interest in developing unlearnable examples (UEs) against visual privacy leaks on the Internet. UEs are training samples added with invisible but unlearnable noise, which have been found can prevent un…
Data Poisoning