Lost in Space Marking
We look at a decision taken early in training a subword tokenizer, namely whether it should be the word-initial token that carries a special mark, or the word-final one. Based on surface-level considerations of efficiency and cohesion, as well as morphological coverage, we find that a Unigram LM tokenizer trained on pre-tokenized English text is better off marking the word-initial token, while one trained on raw text benefits from marking word ends. Our findings generalize across domains.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Meet Your New Client: Writing Reports for AI -- Benchmarking Information Loss in Market Research Deliverables
As organizations adopt retrieval-augmented generation (RAG) for their knowledge management systems (KMS), traditional market research deliverables face new functional demands. While PDF reports and slides have long serve…
Picking watermarks from noise (PWFN): an improved robust watermarking model against intensive distortions
Digital watermarking is the process of embedding secret information by altering images in an undetectable way to the human eye. To increase the robustness of the model, many deep learning-based watermarking methods use t…
DecoderLost in Overlap: Exploring Logit-based Watermark Collision in LLMs
The proliferation of large language models (LLMs) in generating content raises concerns about text copyright. Watermarking methods, particularly logit-based approaches, embed imperceptible identifiers into text to addres…
PhilosophyQuestion AnsweringLoST: Level of Semantics Tokenization for 3D Shapes
Tokenization is a fundamental technique in the generative modeling of various modalities. In particular, it plays a critical role in autoregressive (AR) models, which have recently emerged as a compelling option for 3D g…
Semantic Retrieval3D GenerationLOST: Low-rank and Sparse Pre-training for Large Language Models
While large language models (LLMs) have achieved remarkable performance across a wide range of tasks, their massive scale incurs prohibitive computational and memory costs for pre-training from scratch. Recent studies ha…