Splitting compounds with ngrams
Compound words with unmarked word boundaries are problematic for many tasks in NLP and computational linguistics, including information extraction, machine translation, and syllabification. This paper introduces a simple, proof-of-concept language modeling approach to automatic compound segmentation, as applied to Finnish. This approach utilizes an off-the-shelf morphological analyzer to split training words into their constituent morphemes. A language model is subsequently trained on ngrams composed of morphemes, morpheme boundaries, and word boundaries. Linguistic constraints are then used to weed out phonotactically ill-formed segmentations, thereby allowing the language model to select the best grammatical segmentation. This approach achieves an accuracy of {\textasciitilde}97{\%}.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingMachine TranslationMorphological AnalysisSegmentationSemantic ParsingTranslationSimilar Papers 제목 키워드 기반
A Joint Approach to Compound Splitting and Idiomatic Compound Detection
Applications such as machine translation, speech recognition, and information retrieval require efficient handling of noun compounds as they are one of the possible sources for out-of-vocabulary (OOV) words. In-depth pro…
Information RetrievalMachine TranslationRetrievalspeech-recognition+2Splitting Compounds by Semantic Analogy
Compounding is a highly productive word-formation process in some languages that is often problematic for natural language processing applications. In this paper, we investigate whether distributional semantics in the fo…
Machine TranslationTranslationWord EmbeddingsMulti-Granular Text Encoding for Self-Explaining Categorization
Self-explaining text categorization requires a classifier to make a prediction along with supporting evidence. A popular type of evidence is sub-sequences extracted from the input text which are sufficient for the classi…
Text CategorizationSanskrit Word Segmentation Using Character-level Recurrent and Convolutional Neural Networks
The paper introduces end-to-end neural network models that tokenize Sanskrit by jointly splitting compounds and resolving phonetic merges (Sandhi). Tokenization of Sanskrit depends on local phonetic and distant semantic …
Feature EngineeringLSTM Neural Network for Textual Ngrams
Cognitive neuroscience is the study of how the human brain functions on tasks like decision making, language, perception and reasoning. Deep learning is a class of machine learning algorithms that use neural networks. Th…
Decision MakingDeep Learning