Unsupervised Dependency Graph Network
Recent work has identified properties of pretrained self-attention models that mirror those of dependency parse structures. In particular, some self-attention heads correspond well to individual dependency types. Inspired by these developments, we propose a new competitive mechanism that encourages these attention heads to model different dependency relations. We introduce a new model, the Unsupervised Dependency Graph Network (UDGN), that can induce dependency structures from raw corpora and the masked language modeling task. Experiment results show that UDGN achieves very strong unsupervised dependency parsing performance without gold POS tags and any other external information. The competitive gated heads show a strong correlation with human-annotated dependency types. Furthermore, the UDGN can also achieve competitive performance on masked language modeling and sentence textual similarity tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Dependency ParsingLanguage ModelingLanguage ModellingMasked Language ModelingPOSSentenceUnsupervised Dependency ParsingSimilar Papers 제목 키워드 기반
Unsupervised Dependency Graph Network
Recent work has identified properties of pretrained self-attention models that mirror those of dependency parse structures. In particular, some self-attention heads correspond well to individual dependency types. Inspire…
Dependency ParsingLanguage ModelingLanguage ModellingMasked Language Modeling+3StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling
There are two major classes of natural language grammar -- the dependency grammar that models one-to-one correspondences between words and the constituency grammar that models the assembly of one or several corresponded …
Constituency ParsingDependency ParsingLanguage ModelingLanguage Modelling+2Fill it up: Exploiting partial dependency annotations in a minimum spanning tree parser
Unsupervised models of dependency parsing typically require large amounts of clean, unlabeled data plus gold-standard part-of-speech tags. Adding indirect supervision (e.g. language universals and rules) can help, but we…
Dependency ParsingUnsupervised Topic-Specific Domain Dependency Graphs for Aspect Identification in Sentiment Analysis
Dependency Grammar Induction with a Neural Variational Transition-based Parser
Dependency grammar induction is the task of learning dependency syntax without annotated training data. Traditional graph-based models with global inference achieve state-of-the-art results on this task but they require …
Dependency Grammar InductionVariational Inference