Llamazip: Leveraging LLaMA for Lossless Text Compression and Training Dataset Detection
This work introduces Llamazip, a novel lossless text compression algorithm based on the predictive capabilities of the LLaMA3 language model. Llamazip achieves significant data reduction by only storing tokens that the model fails to predict, optimizing storage efficiency without compromising data integrity. Key factors affecting its performance, including quantization and context window size, are analyzed, revealing their impact on compression ratios and computational requirements. Beyond compression, Llamazip demonstrates the potential to identify whether a document was part of the training dataset of a language model. This capability addresses critical concerns about data provenance, intellectual property, and transparency in language model training.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
LLMZip: Lossless Text Compression using Large Language Models
We provide new estimates of an asymptotic upper bound on the entropy of English using the large language model LLaMA-7B as a predictor for the next token given a window of past tokens. This estimate is significantly smal…
Language ModelingLanguage ModellingLarge Language ModelText CompressionCompression for Better: A General and Stable Lossless Compression Framework
This work focus on how to stabilize and lossless model compression, aiming to reduce model complexity and enhance efficiency without sacrificing performance due to compression errors. A key challenge is effectively lever…
Computational EfficiencyModel CompressionQuantizationLUMI: Tokenizer-Agnostic LLM-Based Lossless Image Compression
Large language model (LLM)-based lossless image compression methods typically represent pixel data through the native text interface of a pretrained model, converting pixel values into token sequences that the LLM proces…
Image CompressionApproximating Human-Like Few-shot Learning with GPT-based Compression
In this work, we conceptualize the learning process as information compression. We seek to equip generative pre-trained models with human-like learning capabilities that enable data compression during inference. We prese…
Data CompressionFew-Shot LearningSemantic SimilaritySemantic Textual Similarity+4ZipNN: Lossless Compression for AI Models
With the growth of model sizes and the scale of their deployment, their sheer size burdens the infrastructure requiring more network and more storage to accommodate these. While there is a vast model compression literatu…
Model Compression