Gradient-Based Subword Tokenization

2000년 도입 · 논문 5편에서 사용

GBST, or Gradient-based Subword Tokenization Module, is a soft gradient-based subword tokenization module that automatically learns latent subword representations from characters in a data-driven fashion. Concretely, GBST enumerates candidate subword blocks and learns to score them in a position-wise fashion using a block scoring network. GBST learns a position-wise soft selection over candidate subword blocks by scoring them with a scoring network. In contrast to prior tokenization-free methods, GBST learns interpretable latent subwords, which enables easy inspection of lexical representations and is more efficient than other byte-based models.

출처: Charformer: Fast Character Transformers via Gradient-based Subword Tokenization

소개 논문: Charformer: Fast Character Transformers via Gradient-based Subword Tokenization