BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training

2024-09-06 · Pavel Chizhov, Catherine Arnett, Elizaveta Korotkova, Ivan P. Yamshchikov

Language models can largely benefit from efficient tokenization. However, they still mostly utilize the classical BPE algorithm, a simple and reliable method. This has been shown to cause such issues as under-trained tokens and sub-optimal compression that may affect the downstream performance. We introduce Picky BPE, a modified BPE algorithm that carries out vocabulary refinement during tokenizer training. Our method improves vocabulary efficiency, eliminates under-trained tokens, and does not compromise text compression. Our experiments show that our method does not reduce the downstream performance, and in several cases improves it.

📄 PDF Abstract BibTeX arXiv:2409.04599

Code (1)

pchizhov/picky_bpe 공식 구현

Tasks

Text Compression

Methods 이 논문이 사용한 방법론

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Optimal Turkish Subword Strategies at Scale: Systematic Evaluation of Data, Vocabulary, Morphology Interplay

2026-02-06 · Duygu Altinok arxiv

Tokenization is a pivotal design choice for neural language modeling in morphologically rich languages (MRLs) such as Turkish, where productive agglutination challenges both vocabulary efficiency and morphological fideli…

Dependency Parsing Sentiment Analysis

Explaining and Mitigating Crosslingual Tokenizer Inequities

2025-10-24 · Catherine Arnett, Tyler A. Chang, Stella Biderman, Benjamin K. Bergen arxiv

The number of tokens it takes to encode parallel text in different languages is known to vary. These disparities are called token premiums. Having high token premiums leads to less throughput during training and increase…

Length-MAX Tokenizer for Language Models

2025-11-25 · Dong Dong, Weijie Su arxiv

We introduce a new tokenizer for language models that minimizes the average tokens per character, thereby reducing the number of tokens needed to represent text during training and to generate text during inference. Our …

graph partitioning

LiteToken: Removing Intermediate Merge Residues From BPE Tokenizers

2026-02-04 · Yike Sun, Haotong Yang, Zhouchen Lin, Muhan Zhang arxiv

Tokenization is fundamental to how language models represent and process text, yet the behavior of widely used BPE tokenizers has received far less study than model architectures and training. In this paper, we investiga…

Training a Tokenizer for Free with Private Federated Learning

2022-03-15 · Eugene Bagdasaryan, Congzheng Song, Rogier Van Dalen, Matt Seigel 외

Federated learning with differential privacy, i.e. private federated learning (PFL), makes it possible to train models on private data distributed across users' devices without harming privacy. PFL is efficient for model…

Federated Learning Language Modelling