Semantic Tokenizer for Enhanced Natural Language Processing
Traditionally, NLP performance improvement has been focused on improving models and increasing the number of parameters. Little attention has been paid to vocabulary optimization. We present a novel tokenizer that uses semantics to drive subword formation. The tokenizer includes a trainer that uses stemming to enhance subword formation. Further optimizations and adaptations are implemented to minimize the number of words that cannot be encoded. The encoder is updated to integrate with the trainer. The tokenizer is implemented as a drop-in replacement for the SentencePiece tokenizer. The new tokenizer more than doubles the number of wordforms represented in the vocabulary. The enhanced vocabulary significantly improves model convergence, quality of word and sentence embeddings. Our experimental results show top performance on two Glue tasks using BERT-base, improving on models more than $20\times$ in size.
Code (0)
등록된 구현이 없습니다.
Tasks
SentenceSentence EmbeddingsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Semantic Tokenizer for Enhanced Natural Language Processing
Traditionally, NLP performance improvement has been focused on improving models and increasing the number of model parameters. NLP vocabulary construction has remained focused on maximizing the number of words represente…
SentenceSentence EmbeddingsEvaluating Persian Tokenizers
Tokenization plays a significant role in the process of lexical analysis. Tokens become the input for other natural language processing tasks, like semantic parsing and language modeling. Natural Language Processing in P…
Language ModelingLanguage ModellingLexical AnalysisSemantic ParsingSupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance
Tokenization remains a fundamental yet underexplored bottleneck in natural language processing, with strategies largely static despite remarkable progress in model architectures. We present SupraTok, a novel tokenization…
SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data
How to boost speech pre-training with textual data is an unsolved problem due to the fact that speech and text are very different modalities with distinct characteristics. In this paper, we propose a cross-modal Speech a…
Language ModelingLanguage Modellingspeech-recognitionSpeech RecognitionHow Can Large Language Models Understand Spatial-Temporal Data?
While Large Language Models (LLMs) dominate tasks like natural language processing and computer vision, harnessing their power for spatial-temporal forecasting remains challenging. The disparity between sequential text a…
Natural Language Understanding