Vaporetto: Efficient Japanese Tokenization Based on Improved Pointwise Linear Classification
This paper proposes an approach to improve the runtime efficiency of Japanese tokenization based on the pointwise linear classification (PLC) framework, which formulates the whole tokenization process as a sequence of linear classification problems. Our approach optimizes tokenization by leveraging the characteristics of the PLC framework and the task definition. Our approach involves (1) composing multiple classifications into array-based operations, (2) efficient feature lookup with memory-optimized automata, and (3) three orthogonal pre-processing methods for reducing actual score calculation. Thus, our approach makes the tokenization speed 5.7 times faster than the current approach based on the same model without decreasing tokenization accuracy. Our implementation is available at https://github.com/daac-tools/vaporetto under the MIT or Apache-2.0 license.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Improving Patent Translation using Bilingual Term Extraction and Re-tokenization for Chinese--Japanese
Unlike European languages, many Asian languages like Chinese and Japanese do not have typographic boundaries in written system. Word segmentation (tokenization) that break sentences down into individual words (tokens) is…
Chinese Word SegmentationMachine TranslationSegmentationTerm Extraction+1Revisiting the Design Issues of Local Models for Japanese Predicate-Argument Structure Analysis
The research trend in Japanese predicate-argument structure (PAS) analysis is shifting from pointwise prediction models with local features to global models designed to search for globally optimal solutions. However, the…
Stochastic Tokenization with a Language Model for Neural Text Classification
For unsegmented languages such as Japanese and Chinese, tokenization of a sentence has a significant impact on the performance of text classification. Sentences are usually segmented with words or subwords by a morpholog…
ClassificationGeneral ClassificationLanguage ModelingLanguage Modelling+5fugashi, a Tool for Tokenizing Japanese in Python
Recent years have seen an increase in the number of large-scale multilingual NLP projects. However, even in such projects, languages with special processing requirements are often excluded. One such language is Japanese.…
Multilingual NLPInconsistent Tokenizations Cause Language Models to be Perplexed by Japanese Grammar
Typical methods for evaluating the performance of language models evaluate their ability to answer questions accurately. These evaluation metrics are acceptable for determining the extent to which language models can und…
Machine TranslationSentence