Sequence Modeling via Segmentations
Segmental structure is a common pattern in many types of sequences such as phrases in human languages. In this paper, we present a probabilistic model for sequences via their segmentations. The probability of a segmented sequence is calculated as the product of the probabilities of all its segments, where each segment is modeled using existing tools such as recurrent neural networks. Since the segmentation of a sequence is usually unknown in advance, we sum over all valid segmentations to obtain the final probability for the sequence. An efficient dynamic programming algorithm is developed for forward and backward computations without resorting to any approximation. We demonstrate our approach on text segmentation and speech recognition tasks. In addition to quantitative results, we also show that our approach can discover meaningful segments in their respective application contexts.
Code (2)
Tasks
Segmentationspeech-recognitionSpeech RecognitionText SegmentationvalidSimilar Papers 제목 키워드 기반
Training Hybrid Language Models by Marginalizing over Segmentations
In this paper, we study the problem of hybrid language modeling, that is using models which can predict both characters and larger units such as character ngrams or words. Using such models, multiple potential segmentati…
Language ModelingLanguage ModellingSOMBRERO: Measuring and Steering Boundary Placement in End-to-End Hierarchical Sequence Models
Hierarchical sequence models replace fixed tokenization with learned segmentations that compress long byte sequences for efficient autoregressive modeling. While recent end-to-end methods can learn meaningful boundaries …
Lattice-Based Transformer Encoder for Neural Machine Translation
Neural machine translation (NMT) takes deterministic sequences for source representations. However, either word-level or subword-level segmentations have multiple choices to split a source sequence with different word se…
DiversityMachine TranslationNMTTranslationSequential Modeling Enables Scalable Learning for Large Vision Models
We introduce a novel sequential modeling approach which enables learning a Large Vision Model (LVM) without making use of any linguistic data. To do this, we define a common format, "visual sentences", in which we can re…
DiversityJoint 3D Scene Reconstruction and Class Segmentation
Both image segmentation and dense 3D modeling from images represent an intrinsically ill-posed problem. Strong regularizers are therefore required to constrain the solutions from being 'too noisy'. Unfortunately, these p…
3D Reconstruction3D Scene ReconstructionImage SegmentationSegmentation+1