A Masked Segmental Language Model for Unsupervised Natural Language Segmentation
Segmentation remains an important preprocessing step both in languages where "words" or other important syntactic/semantic units (like morphemes) are not clearly delineated by white space, as well as when dealing with continuous speech data, where there is often no meaningful pause between words. Near-perfect supervised methods have been developed for use in resource-rich languages such as Chinese, but many of the world's languages are both morphologically complex, and have no large dataset of "gold" segmentations into meaningful units. To solve this problem, we propose a new type of Segmental Language Model (Sun and Deng, 2018; Kawakami et al., 2019; Wang et al., 2021) for use in both unsupervised and lightly supervised segmentation tasks. We introduce a Masked Segmental Language Model (MSLM) built on a span-masking transformer architecture, harnessing the power of a bi-directional masked modeling context and attention. In a series of experiments, our model consistently outperforms Recurrent SLMs on Chinese (PKU Corpus) in segmentation quality, and performs similarly to the Recurrent model on English (PTB). We conclude by discussing the different challenges posed in segmenting phonemic-type writing systems.
Code (1)
Tasks
Language ModelingLanguage ModellingSegmentationSimilar Papers 제목 키워드 기반
A Masked Segmental Language Model for Unsupervised Natural Language Segmentation
We introduce a Masked Segmental Language Model (MSLM) for joint language modeling and unsupervised segmentation. While near-perfect supervised methods have been developed for segmenting human-like linguistic units in res…
Language ModelingLanguage ModellingSegmentationUnsupervised Neural Word Segmentation for Chinese via Segmental Language Modeling
Previous traditional approaches to unsupervised Chinese word segmentation (CWS) can be roughly classified into discriminative and generative models. The former uses the carefully designed goodness measures for candidate …
Chinese Word SegmentationDecoderLanguage ModelingLanguage Modelling+1Multilingual unsupervised sequence segmentation transfers to extremely low-resource languages
We show that unsupervised sequence-segmentation performance can be transferred to extremely low-resource languages by pre-training a Masked Segmental Language Model (Downey et al., 2021) multilingually. Further, we show …
Language ModelingLanguage ModellingSegmentationMultilingual unsupervised sequence segmentation transfers to extremely low-resource languages
We show that unsupervised sequence-segmentation performance can be transferred to extremely low-resource languages by pre-training a Masked Segmental Language Model (Downey et al., 2021) multilingually. Further, we show …
Language ModelingLanguage ModellingUnsupervised Speech Recognition via Segmental Empirical Output Distribution Matching
We consider the problem of training speech recognition systems without using any labeled data, under the assumption that the learner can only access to the input utterances and a phoneme language model estimated from a n…
Language ModelingLanguage ModellingPhoneme Recognitionspeech-recognition+2