Lossless Data Compression with Transformer
Transformers have replaced long-short term memory and other recurrent neural networks variants in sequence modeling. It achieves state-of-the-art performance on a wide range of tasks related to natural language processing, including language modeling, machine translation, and sentence representation. Lossless compression is another problem that can benefit from better sequence models. It is closely related to the problem of online learning of language models. But, despite this ressemblance, it is an area where purely neural network based methods have not yet reached the compression ratio of state-of-the-art algorithms. In this paper, we propose a Transformer based lossless compression method that match the best compression ratio for text. Our approach is purely based on neural networks and does not rely on hand-crafted features as other lossless compression algorithms. We also provide a thorough study of the impact of the different components of the Transformer and its training on the compression ratio.
Code (0)
등록된 구현이 없습니다.
Tasks
Data CompressionLanguage ModelingLanguage ModellingMachine TranslationSentenceTranslationSimilar Papers 제목 키워드 기반
An Enhanced Text Compression Approach Using Transformer-based Language Models
Text compression shrinks textual data while keeping crucial information, eradicating constraints on storage, bandwidth, and computational efficacy. The integration of lossless compression techniques with transformer-base…
de-enText CompressionA Fast Transformer-based General-Purpose Lossless Compressor
Deep-learning-based compressor has received interests recently due to much improved compression ratio. However, modern approaches suffer from long execution time. To ease this problem, this paper targets on cutting down …
Deep LearningAlphaZip: Neural Network-Enhanced Lossless Text Compression
Data compression continues to evolve, with traditional information theory methods being widely used for compressing text, images, and videos. Recently, there has been growing interest in leveraging Generative AI for pred…
BenchmarkingData CompressionLanguage ModelingLanguage Modelling+2Lossless Compression: A New Benchmark for Time Series Model Evaluation
The evaluation of time series models has traditionally focused on four canonical tasks: forecasting, imputation, anomaly detection, and classification. While these tasks have driven significant progress, they primarily a…
Anomaly DetectionDeep Lossy Plus Residual Coding for Lossless and Near-lossless Image Compression
Lossless and near-lossless image compression is of paramount importance to professional users in many technical fields, such as medicine, remote sensing, precision engineering and scientific research. But despite rapidly…
Image Compression