Pruned BPE: Post-training Visibility Pruning and Token Reallocation for Byte Pair Encoding

2026-08-01 · Kenny Shao arxiv

Byte Pair Encoding (BPE) is widely used for subword tokenization, but standard BPE exposes every learned merge token to the downstream model, including tokens that mainly serve as intermediate construction units and rarely appear in the final encoded corpus. This paper proposes Pruned BPE, a post-training visibility-pruning and token-reallocation method that separates merge construction from model-visible vocabulary selection. After standard BPE training, tokens are evaluated by final exposure. Low-exposure tokens are retained as internal-only merge nodes, while their visible vocabulary slots are reassigned to better-exposed candidates learned through resumed training. During encoding, internal-only tokens are recursively expanded into visible descendants while the original BPE merge order is preserved. Experiments on two non-overlapping English- and Chinese-dominated corpora and their combination show that Pruned BPE consistently reduces encoded length relative to Standard BPE at the same training corpus, evaluation corpus, and model-visible vocabulary size. At a 40% exposure threshold, the reduction is approximately 0.27%--0.36% on same-corpus evaluations. In a vocabulary-only evaluation using a shared exact minimum-token dynamic-programming encoder, Pruned BPE retains an advantage of approximately 0.23%--0.31%, indicating that the improvement arises from a more efficient visible vocabulary. These gains represent a meaningful fraction of the approximately 1.5%--3.8% marginal reduction that would otherwise require adding another 2K Standard BPE tokens. Qualitative analysis shows that internal-only tokens include reusable English fragments, Chinese components, partial UTF-8 byte sequences, and structured-text fragments. The results indicate that post-training visibility pruning can improve BPE vocabulary efficiency without increasing the vocabulary exposed to the language model.

📄 PDF Abstract BibTeX arXiv:2608.00837

Code (0)

등록된 구현이 없습니다.

P$^2$ Law: Scaling Law for Post-Training After Model Pruning

2024-11-15 · Xiaodong Chen, Yuxuan Hu, Xiaokang Zhang, Yanling Wang 외

Pruning has become a widely adopted technique for reducing the hardware requirements of large language models (LLMs). To recover model performance after pruning, post-training is commonly employed to mitigate the resulti…

On the Limits of Layer Pruning for Generative Reasoning in Large Language Models

2026-02-02 · Safal Shrestha, Anubhav Shrestha, Aadim Nepal, Minwu Kim 외 arxiv

Recent work has shown that layer pruning can effectively compress large language models (LLMs) while retaining strong performance on classification benchmarks, often with little or no finetuning. In contrast, generative …

IPCV: Information-Preserving Compression for MLLM Visual Encoders

2025-12-21 · Yuan Chen, Zichen Wen, Yuzhou Wu, Xuyang Liu 외 arxiv

Multimodal Large Language Models (MLLMs) deliver strong vision-language performance but at high computational cost, driven by numerous visual tokens processed by the Vision Transformer (ViT) encoder. Existing token pruni…

TraceNAS: Zero-shot LLM Pruning via Gradient Trace Correlation

2026-02-02 · Prajna G. Malettira, Manish Nagaraj, Arjun Roy, Shubham Negi 외 arxiv

Structured pruning is essential for efficient deployment of Large Language Models (LLMs). The varying sensitivity of LLM sub-blocks to pruning necessitates the identification of optimal non-uniformly pruned models. Exist…

Neural Architecture Search

Distilling Token-Pruned Pose Transformer for 2D Human Pose Estimation

2023-04-12 · Feixiang Ren

Human pose estimation has seen widespread use of transformer models in recent years. Pose transformers benefit from the self-attention map, which captures the correlation between human joint tokens and the image. However…

2D Human Pose Estimation Pose Estimation