Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale
A syntactic language model (SLM) incrementally generates a sentence with its syntactic tree in a left-to-right manner. We present Generative Pretrained Structured Transformers (GPST), an unsupervised SLM at scale capable of being pre-trained from scratch on raw texts with high parallelism. GPST circumvents the limitations of previous SLMs such as relying on gold trees and sequential training. It consists of two components, a usual SLM supervised by a uni-directional language modeling loss, and an additional composition model, which induces syntactic parse trees and computes constituent representations, supervised by a bi-directional language modeling loss. We propose a representation surrogate to enable joint parallel training of the two models in a hard-EM fashion. We pre-train GPST on OpenWebText, a corpus with $9$ billion tokens, and demonstrate the superiority of GPST over GPT-2 with a comparable size in numerous tasks covering both language understanding and language generation. Meanwhile, GPST also significantly outperforms existing unsupervised SLMs on left-to-right grammar induction, while holding a substantial acceleration on training.
Code (2)
Tasks
Constituency Grammar InductionLanguage ModelingLanguage ModellingNatural Language InferenceSentenceText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Unsupervised Learning of Syntactic Structure with Invertible Neural Projections
Unsupervised learning of syntactic structure is typically performed using generative models with discrete latent variables and multinomial parameters. In most cases, these models have not leveraged continuous word repres…
Constituency Grammar InductionDependency ParsingPOSUnsupervised Dependency ParsingExploiting Inductive Bias in Transformers for Unsupervised Disentanglement of Syntax and Semantics with VAEs
We propose a generative model for text generation, which exhibits disentangled latent representations of syntax and semantics. Contrary to previous work, this model does not need syntactic information such as constituenc…
DecoderDisentanglementInductive BiasText GenerationHow much pretraining data do language models need to learn syntax?
Transformers-based pretrained language models achieve outstanding results in many well-known NLU benchmarks. However, while pretraining methods are very convenient, they are expensive in terms of time and resources. This…
Dependency ParsingParaphrase IdentificationPart-Of-Speech TaggingEmpirical Study of Transformers for Source Code
Initially developed for natural language processing (NLP), Transformers are now widely used for source code processing, due to the format similarity between source code and text. In contrast to natural language, source c…
Bug fixingCode CompletionGrouping First, Attending Smartly: Training-Free Acceleration for Diffusion Transformers
Diffusion-based Transformers have demonstrated impressive generative capabilities, but their high computational costs hinder practical deployment, for example, generating an $8192\times 8192$ image can take over an hour …
GPUVideo Generation