Universal Dependencies according to BERT: both more specific and more general
This work focuses on analyzing the form and extent of syntactic abstraction captured by BERT by extracting labeled dependency trees from self-attentions. Previous work showed that individual BERT heads tend to encode particular dependency relation types. We extend these findings by explicitly comparing BERT relations to Universal Dependencies (UD) annotations, showing that they often do not match one-to-one. We suggest a method for relation identification and syntactic tree construction. Our approach produces significantly more consistent dependency trees than previous work, showing that it better explains the syntactic abstractions in BERT. At the same time, it can be successfully applied with only a minimal amount of supervision and generalizes well across languages.
Code (2)
Tasks
RelationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Køpsala: Transition-Based Graph Parsing via Efficient Training and Effective Encoding
We present K{\o}psala, the Copenhagen-Uppsala system for the Enhanced Universal Dependencies Shared Task at IWPT 2020. Our system is a pipeline consisting of off-the-shelf models for everything but enhanced graph parsing…
SentenceK\opsala: Transition-Based Graph Parsing via Efficient Training and Effective Encoding
We present K{\o}psala, the Copenhagen-Uppsala system for the Enhanced Universal Dependencies Shared Task at IWPT 2020. Our system is a pipeline consisting of off-the-shelf models for everything but enhanced graph parsing…
SentencePortiLexicon-UD: a Portuguese Lexical Resource according to Universal Dependencies Model
This paper presents PortiLexicon-UD, a large and freely available lexicon for Portuguese delivering morphosyntactic information according to the Universal Dependencies model. This lexical resource includes part of speech…
LEMMAPOSTAGUniversal dependencies for Uyghur
The Universal Dependencies (UD) Project seeks to build a cross-lingual studies of treebanks, linguistic structures and parsing. Its goal is to create a set of multilingual harmonized treebanks that are designed according…
Cross-Lingual Transfer75 Languages, 1 Model: Parsing Universal Dependencies Universally
We present UDify, a multilingual multi-task model capable of accurately predicting universal part-of-speech, morphological features, lemmas, and dependency trees simultaneously for all 124 Universal Dependencies treebank…
Dependency ParsingmodelZero-Shot Learning