Have Attention Heads in BERT Learned Constituency Grammar?
With the success of pre-trained language models in recent years, more and more researchers focus on opening the "black box" of these models. Following this interest, we carry out a qualitative and quantitative analysis of constituency grammar in attention heads of BERT and RoBERTa. We employ the syntactic distance method to extract implicit constituency grammar from the attention weights of each head. Our results show that there exist heads that can induce some grammar types much better than baselines, suggesting that some heads act as a proxy for constituency grammar. We also analyze how attention heads' constituency grammar inducing (CGI) ability changes after fine-tuning with two kinds of tasks, including sentence meaning similarity (SMS) tasks and natural language inference (NLI) tasks. Our results suggest that SMS tasks decrease the average CGI ability of upper layers, while NLI tasks increase it. Lastly, we investigate the connections between CGI ability and natural language understanding ability on QQP and MNLI tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Natural Language InferenceNatural Language UnderstandingQQPSentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Better heads do not guarantee better binarized constituency parsing
We revisit punctuation-aware tree binarization for constituency parsing and ask whether dependency-induced headedness improves binary parser supervision. Although learned heads substantially outperform rule-based heads i…
Constituency ParsingHeads-up! Unsupervised Constituency Parsing via Self-Attention Heads
Transformer-based pre-trained language models (PLMs) have dramatically improved the state of the art in NLP across many tasks. This has led to substantial interest in analyzing the syntactic knowledge PLMs learn. Previou…
Constituency ParsingDo Transformers Parse while Predicting the Masked Word?
Pre-trained language models have been shown to encode linguistic structures, e.g. dependency and constituency parse trees, in their embeddings while being trained on unsupervised loss functions like masked language model…
Constituency ParsingLanguage ModelingLanguage ModellingMasked Language ModelingConvBERT: Improving BERT with Span-based Dynamic Convolution
Pre-trained language models like BERT and its variants have recently achieved impressive performance in various natural language understanding tasks. However, BERT heavily relies on the global self-attention block and th…
Natural Language UnderstandingAn Empirical Study for Vietnamese Constituency Parsing with Pre-training
In this work, we use a span-based approach for Vietnamese constituency parsing. Our method follows the self-attention encoder architecture and a chart decoder using a CKY-style inference algorithm. We present analyses of…
Constituency ParsingDecoderVietnamese DatasetsVietnamese Parsing