Rethinking Self-Attention: Towards Interpretability in Neural Parsing
Attention mechanisms have improved the performance of NLP tasks while allowing models to remain explainable. Self-attention is currently widely used, however interpretability is difficult due to the numerous attention distributions. Recent work has shown that model representations can benefit from label-specific information, while facilitating interpretation of predictions. We introduce the Label Attention Layer: a new form of self-attention where attention heads represent labels. We test our novel layer by running constituency and dependency parsing experiments and show our new model obtains new state-of-the-art results for both tasks on both the Penn Treebank (PTB) and Chinese Treebank. Additionally, our model requires fewer self-attention layers compared to existing work. Finally, we find that the Label Attention heads learn relations between syntactic categories and show pathways to analyze errors.
Code (2)
Tasks
Constituency ParsingDependency ParsingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
UC Davis at SemEval-2019 Task 1: DAG Semantic Parsing with Attention-based Decoder
We present an encoder-decoder model for semantic parsing with UCCA SemEval 2019 Task 1. The encoder is a Bi-LSTM and the decoder uses recursive self-attention. The proposed model alleviates challenges and feature enginee…
DecoderFeature EngineeringSemantic ParsingUnsupervised and Few-shot Parsing from Pretrained Language Models
Pretrained language models are generally acknowledged to be able to encode syntax [Tenney et al., 2019, Jawahar et al., 2019, Hewitt and Manning, 2019]. In this article, we propose UPOA, an Unsupervised constituent Parsi…
Language ModellingSCLIP: Rethinking Self-Attention for Dense Vision-Language Inference
Recent advances in contrastive language-image pretraining (CLIP) have demonstrated strong capabilities in zero-shot classification by aligning visual representations with target text embeddings in an image level. However…
SegmentationSemantic Segmentationzero-shot-classificationZero-Shot Learning+1An Imitation Learning Approach to Unsupervised Parsing
Recently, there has been an increasing interest in unsupervised parsers that optimize semantically oriented objectives, typically using reinforcement learning. Unfortunately, the learned trees often do not match actual s…
Imitation LearningLanguage ModelingLanguage ModellingNatural Language Inference+1Multi-Scale Dual-Branch Fully Convolutional Network for Hand Parsing
Recently, fully convolutional neural networks (FCNs) have shown significant performance in image parsing, including scene parsing and object parsing. Different from generic object parsing tasks, hand parsing is more chal…
Multi-class ClassificationScene Parsing