A Masked Segmental Language Model for Unsupervised Natural Language Segmentation
We introduce a Masked Segmental Language Model (MSLM) for joint language modeling and unsupervised segmentation. While near-perfect supervised methods have been developed for segmenting human-like linguistic units in resource-rich languages such as Chinese, many of the world's languages are both morphologically complex, and have no large dataset of ``gold'' segmentations for supervised training. Segmental Language Models offer a unique approach by conducting unsupervised segmentation as the byproduct of a neural language modeling objective. However, current SLMs are limited in their scalability due to their recurrent architecture. We propose a new type of SLM for use in both unsupervised and lightly supervised segmentation tasks. The MSLM is built on a span-masking transformer architecture, harnessing a masked bidirectional modeling context and attention, as well as adding the potential for model scalability. In a series of experiments, our model outperforms the segmentation quality of recurrent SLMs on Chinese, and performs similarly to the recurrent model on English.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingSegmentationSimilar Papers 제목 키워드 기반
A Masked Segmental Language Model for Unsupervised Natural Language Segmentation
Segmentation remains an important preprocessing step both in languages where "words" or other important syntactic/semantic units (like morphemes) are not clearly delineated by white space, as well as when dealing with co…
Language ModelingLanguage ModellingSegmentationUnsupervised Neural Word Segmentation for Chinese via Segmental Language Modeling
Previous traditional approaches to unsupervised Chinese word segmentation (CWS) can be roughly classified into discriminative and generative models. The former uses the carefully designed goodness measures for candidate …
Chinese Word SegmentationDecoderLanguage ModelingLanguage Modelling+1Multilingual unsupervised sequence segmentation transfers to extremely low-resource languages
We show that unsupervised sequence-segmentation performance can be transferred to extremely low-resource languages by pre-training a Masked Segmental Language Model (Downey et al., 2021) multilingually. Further, we show …
Language ModelingLanguage ModellingSegmentationMultilingual unsupervised sequence segmentation transfers to extremely low-resource languages
We show that unsupervised sequence-segmentation performance can be transferred to extremely low-resource languages by pre-training a Masked Segmental Language Model (Downey et al., 2021) multilingually. Further, we show …
Language ModelingLanguage ModellingUnsupervised Speech Recognition via Segmental Empirical Output Distribution Matching
We consider the problem of training speech recognition systems without using any labeled data, under the assumption that the learner can only access to the input utterances and a phoneme language model estimated from a n…
Language ModelingLanguage ModellingPhoneme Recognitionspeech-recognition+2