Papers Text Compression
“Text Compression” 태그가 달린 논문 43편 · 필터 해제
TokAlign: Efficient Vocabulary Adaptation via Token Alignment
Tokenization serves as a foundational step for Large Language Models (LLMs) to process text. In new domains or languages, the inefficiency of the tokenizer will slow down the training and generation of LLM. The mismatch …
SentenceText CompressionTransfer LearningBeyond Text Compression: Evaluating Tokenizers Across Scales
The choice of tokenizer can profoundly impact language model performance, yet accessible and reliable evaluations of tokenizer quality remain an open challenge. Inspired by scaling consistency, we show that smaller model…
Language ModelingLanguage ModellingText CompressionMeasuring Information Distortion in Hierarchical Ultra long Novel Generation:The Optimal Expansion Ratio
Writing novels with Large Language Models (LLMs) raises a critical question: how much human-authored outline is necessary to generate high-quality million-word novels? While frameworks such as DOME, Plan&Write, and Long …
Text CompressionHypernym Mercury: Token Optimization Through Semantic Field Constriction And Reconstruction From Hypernyms. A New Text Compression Method
Compute optimization using token reduction of LLM prompts is an emerging task in the fields of NLP and next generation, agentic AI. In this white paper, we introduce a novel (patent pending) text representation scheme an…
Semantic CompressionSemantic SimilaritySemantic Textual SimilarityText Compression+1Lossless Compression of Large Language Model-Generated Text via Next-Token Prediction
As large language models (LLMs) continue to be deployed and utilized across domains, the volume of LLM-generated data is growing rapidly. This trend highlights the increasing importance of effective and lossless compress…
Language ModelingLanguage ModellingLarge Language ModelText CompressionText Compression for Efficient Language Generation
We challenge the prevailing assumption that LLMs must rely fully on sub-word tokens for high-quality text generation. To this end, we propose the "Generative Pretrained Thoughtformer" (GPTHF), a hierarchical transformer …
Language ModelingLanguage ModellingSentenceSentence Embeddings+2Scaling Multi-Document Event Summarization: Evaluating Compression vs. Full-Text Approaches
Automatically summarizing large text collections is a valuable tool for document research, with applications in journalism, academic research, legal work, and many other fields. In this work, we contrast two classes of s…
Document SummarizationMulti-Document SummarizationRetrievalText CompressionAssessing Human Editing Effort on LLM-Generated Texts via Compression-Based Edit Distance
Assessing the extent of human edits on texts generated by Large Language Models (LLMs) is crucial to understanding the human-AI interactions and improving the quality of automated text generation systems. Existing edit d…
Computational EfficiencyText CompressionText GenerationL3TC: Leveraging RWKV for Learned Lossless Low-Complexity Text Compression
Learning-based probabilistic models can be combined with an entropy coder for data compression. However, due to the high complexity of learning-based models, their practical application as text compressors has been large…
Data CompressionText CompressionAn Enhanced Text Compression Approach Using Transformer-based Language Models
Text compression shrinks textual data while keeping crucial information, eradicating constraints on storage, bandwidth, and computational efficacy. The integration of lossless compression techniques with transformer-base…
de-enText CompressionIntellectSeeker: A Personalized Literature Management System with the Probabilistic Model and Large Language Model
Faced with the burgeoning volume of academic literature, researchers often need help with uncertain article quality and mismatches in term searches using traditional academic engines. We introduce IntellectSeeker, an inn…
ArticlesFew-Shot LearningLanguage ModelingLanguage Modelling+4Theoretical Analysis of Byte-Pair Encoding
Byte-Pair Encoding (BPE) is a widely used method for subword tokenization, with origins in grammar-based text compression. It is employed in a variety of language processing tasks such as machine translation or large lan…
Language ModelingLanguage ModellingLarge Language ModelMachine Translation+1FineZip : Pushing the Limits of Large Language Models for Practical Lossless Text Compression
While the language modeling objective has been shown to be deeply connected with compression, it is surprising that modern LLMs are not employed in practical text compression systems. In this paper, we provide an in-dept…
Language ModelingLanguage ModellingMemorizationText CompressionAlphaZip: Neural Network-Enhanced Lossless Text Compression
Data compression continues to evolve, with traditional information theory methods being widely used for compressing text, images, and videos. Recently, there has been growing interest in leveraging Generative AI for pred…
BenchmarkingData CompressionLanguage ModelingLanguage Modelling+2BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training
Language models can largely benefit from efficient tokenization. However, they still mostly utilize the classical BPE algorithm, a simple and reliable method. This has been shown to cause such issues as under-trained tok…
Text CompressionXCompress: LLM assisted Python-based text compression toolkit
This study introduces XCompress, a Python-based tool for effectively utilizing various compression algorithms. XCompress offers manual, brute force, and Large Language Model (LLM) methods to determine the most suitable a…
BenchmarkingLanguage ModelingLanguage ModellingLarge Language Model+1Recurrent Context Compression: Efficiently Expanding the Context Window of LLM
To extend the context length of Transformer-based large language models (LLMs) and improve comprehension capabilities, we often face limitations due to computational resources and bounded memory storage capacity. This wo…
Long-Context UnderstandingQuestion AnsweringRetrievalText CompressionVariational Bayesian Methods for a Tree-Structured Stick-Breaking Process Mixture of Gaussians by Application of the Bayes Codes for Context Tree Models
The tree-structured stick-breaking process (TS-SBP) mixture model is a non-parametric Bayesian model that can represent tree-like hierarchical structures among the mixture components. For TS-SBP mixture models, only a Ma…
Computational EfficiencyText CompressionLLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
This paper focuses on task-agnostic prompt compression for better generalizability and efficiency. Considering the redundancy in natural language, existing approaches compress prompts by removing tokens or lexical units …
GSM8KLanguage ModellingText Compressiontoken-classification+1Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance
Despite it being the cornerstone of BPE, the most common tokenization algorithm, the importance of compression in the tokenization process is still unclear. In this paper, we argue for the theoretical importance of compr…
Language ModelingLanguage ModellingText Compression