Unsupervised Tokenization Learning
In the presented study, we discover that the so-called "transition freedom" metric appears superior for unsupervised tokenization purposes in comparison to statistical metrics such as mutual information and conditional probability, providing F-measure scores in range from 0.71 to 1.0 across explored multilingual corpora. We find that different languages require different offshoots of that metric (such as derivative, variance, and "peak values") for successful tokenization. Larger training corpora do not necessarily result in better tokenization quality, while compressing the models by eliminating statistically weak evidence tends to improve performance. The proposed unsupervised tokenization technique provides quality better than or comparable to lexicon-based ones, depending on the language.
Code (0)
등록된 구현이 없습니다.
Tasks
Text SegmentationSimilar Papers 제목 키워드 기반
Self-tuning hyper-parameters for unsupervised cross-lingual tokenization
We explore the possibility of meta-learning for the language-independent unsupervised tokenization problem for English, Russian, and Chinese. We implement the meta-learning approach for automatic determination of hyper-p…
Meta-LearningA Tokenization System for the Kurdish Language
Tokenization is one of the essential and fundamental tasks in natural language processing. Despite the recent advances in applying unsupervised statistical methods for this task, every language with its writing system an…
A Spitting Image: Modular Superpixel Tokenization in Vision Transformers
Vision Transformer (ViT) architectures traditionally employ a grid-based approach to tokenization independent of the semantic content of an image. We propose a modular superpixel tokenization strategy which decouples tok…
MorphPiece : A Linguistic Tokenizer for Large Language Models
Tokenization is a critical part of modern NLP pipelines. However, contemporary tokenizers for Large Language Models are based on statistical analysis of text corpora, without much consideration to the linguistic features…
Language ModelingLanguage ModellingMaster Thesis: Neural Sign Language Translation by Learning Tokenization
In this thesis, we propose a multitask learning based method to improve Neural Sign Language Translation (NSLT) consisting of two parts, a tokenization layer and Neural Machine Translation (NMT). The tokenization part fo…
Domain AdaptationMachine TranslationNMTSign Language Translation+3