Juman++: A Morphological Analysis Toolkit for Scriptio Continua
We present a three-part toolkit for developing morphological analyzers for languages without natural word boundaries. The first part is a C++11/14 lattice-based morphological analysis library that uses a combination of linear and recurrent neural net language models for analysis. The other parts are a tool for exposing problems in the trained model and a partial annotation tool. Our morphological analyzer of Japanese achieves new SOTA on Jumandic-based corpora while being 250 times faster than the previous one. We also perform a small experiment and quantitive analysis and experience of using development tools. All components of the toolkit is open source and available under a permissive Apache 2 License.
Code (1)
Tasks
Art AnalysisLanguage ModelingLanguage ModellingMorphological AnalysisPart-Of-Speech TaggingSimilar Papers 제목 키워드 기반
TURJUMAN: A Public Toolkit for Neural Arabic Machine Translation
We present TURJUMAN, a neural toolkit for translating from 20 languages into Modern Standard Arabic (MSA). TURJUMAN exploits the recently-introduced text-to-text Transformer AraT5 model, endowing it with a powerful abili…
Machine TranslationSemantic SimilaritySemantic Textual SimilarityTranslationA Finite-state Morphological Analyser for Tuvan
{\textasciitilde}This paper describes the development of free/open-source finite-state morphological transducers for Tuvan, a Turkic language spoken in and around the Tuvan Republic in Russia. The finite-state toolkit us…
Jumanji: a Diverse Suite of Scalable Reinforcement Learning Environments in JAX
Open-source reinforcement learning (RL) environments have played a crucial role in driving progress in the development of AI algorithms. In modern RL research, there is a need for simulated environments that are performa…
Decision Makingreinforcement-learningReinforcement Learning (RL)Spatial Pathomics Toolkit for Quantitative Analysis of Podocyte Nuclei with Histology and Spatial Transcriptomics Data in Renal Pathology
Podocytes, specialized epithelial cells that envelop the glomerular capillaries, play a pivotal role in maintaining renal health. The current description and quantification of features on pathology slides are limited, pr…
Semantic Segmentationwhole slide imagesHow do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese
This paper investigates the effect of tokenizers on the downstream performance of pretrained language models (PLMs) in scriptio continua languages where no explicit spaces exist between words, using Japanese as a case st…