Text Segmentation based on Semantic Word Embeddings
We explore the use of semantic word embeddings in text segmentation algorithms, including the C99 segmentation algorithm and new algorithms inspired by the distributed word vector representation. By developing a general framework for discussing a class of segmentation objectives, we study the effectiveness of greedy versus exact optimization approaches and suggest a new iterative refinement technique for improving the performance of greedy strategies. We compare our results to known benchmarks, using known metrics. We demonstrate state-of-the-art performance for an untrained method with our Content Vector Segmentation (CVS) on the Choi test set. Finally, we apply the segmentation procedure to an in-the-wild dataset consisting of text extracted from scholarly articles in the arXiv.org database.
Code (1)
Tasks
ArticlesSegmentationText SegmentationWord EmbeddingsSimilar Papers 제목 키워드 기반
Context-aware Feature Generation for Zero-shot Semantic Segmentation
Existing semantic segmentation models heavily rely on dense pixel-wise annotations. To reduce the annotation pressure, we focus on a challenging task named zero-shot semantic segmentation, which aims to segment unseen ob…
SegmentationSemantic SegmentationWord EmbeddingsZero Shot Segmentation+1Parsimonious Morpheme Segmentation with an Application to Enriching Word Embeddings
Traditionally, many text-mining tasks treat individual word-tokens as the finest meaningful semantic granularity. However, in many languages and specialized corpora, words are composed by concatenating semantically meani…
Language ModelingLanguage ModellingSegmentationWord EmbeddingsA Pilot Study for Chinese SQL Semantic Parsing
The task of semantic parsing is highly useful for dialogue and question answering systems. Many datasets have been proposed to map natural language text into SQL, among which the recent Spider dataset provides cross-doma…
Cross-Lingual Word EmbeddingsQuestion AnsweringSegmentationSemantic Parsing+3Word-Context Character Embeddings for Chinese Word Segmentation
Neural parsers have benefited from automatically labeled data via dependency-context word embeddings. We investigate training character embeddings on a word-based context in a similar way, showing that the simple method …
Chinese Word SegmentationDomain AdaptationSegmentationWord EmbeddingsA Trie-Structured Bayesian Model for Unsupervised Morphological Segmentation
In this paper, we introduce a trie-structured Bayesian model for unsupervised morphological segmentation. We adopt prior information from different sources in the model. We use neural word embeddings to discover words th…
SegmentationWord Embeddings