XCompress: LLM assisted Python-based text compression toolkit
This study introduces XCompress, a Python-based tool for effectively utilizing various compression algorithms. XCompress offers manual, brute force, and Large Language Model (LLM) methods to determine the most suitable algorithm based on the type of text data. Its modular structure allows easy addition of new algorithms and includes functions for benchmarking and result comparison. Tests on diverse text types demonstrate the efficacy of the LLM-assisted Compression Selection Model (CSM). With XCompress, users can determine the most suitable method for their files. Additionally, in academic research, they can easily compare different methods without needing any scripting or programming language.
Code (1)
Tasks
BenchmarkingLanguage ModelingLanguage ModellingLarge Language ModelText CompressionSimilar Papers 제목 키워드 기반
500xCompressor: Generalized Prompt Compression for Large Language Models
Prompt compression is crucial for enhancing inference speed, reducing costs, and improving user experience. However, current methods face challenges such as low compression ratios and potential data leakage during evalua…
Language ModellingLarge Language ModelQuestion AnsweringMixCompress: Mixture of Experts for Variable Rate Learned Image Compression
Learned image compression (LIC) is bottlenecked by the need to store independent models for each rate-distortion operating point. Existing variable bit-rate (VBR) methods aim to reduce this overhead via dense parameter m…
Image CompressionBDI-Kit Demo: A Toolkit for Programmable and Conversational Data Harmonization
Data harmonization remains a major bottleneck for integrative analysis due to heterogeneity in schemas, value representations, and domain-specific conventions. BDI-Kit provides an extensible toolkit for schema and value …
mwetoolkit-lib: Adaptation of the mwetoolkit as a Python Library and an Application to MWE-based Document Clustering
This paper introduces the mwetoolkit-lib, an adaptation of the mwetoolkit as a python library. The original toolkit performs the extraction and identification of multiword expressions (MWEs) in large text bases through t…
A Flexible, Extensible Software Framework for Neural Net Compression
We propose a software framework based on ideas of the Learning-Compression algorithm , that allows one to compress any neural network by different compression mechanisms (pruning, quantization, low-rank, etc.). By design…
Quantization