홈 › Papers

What's so special about BERT's layers? A closer look at the NLP pipeline in monolingual and multilingual models

2020-04-14 · Findings of the Association for Computational Linguistics 2020 · Wietse de Vries, Andreas van Cranenburgh, Malvina Nissim

Peeking into the inner workings of BERT has shown that its layers resemble the classical NLP pipeline, with progressively more complex tasks being concentrated in later layers. To investigate to what extent these results also hold for a language other than English, we probe a Dutch BERT-based model and the multilingual BERT model for Dutch NLP tasks. In addition, through a deeper analysis of part-of-speech tagging, we show that also within a given task, information is spread over different parts of the network and the pipeline might not be as neat as it seems. Each layer has different specialisations, so that it may be more useful to combine information from different layers, instead of selecting a single one based on the best overall performance.

📄 PDF Abstract BibTeX arXiv:2004.06499

Code (2)

wietsedv/bertje 공식 구현 tf

wietsedv/bertje/tree/master/probing 공식 구현 pytorch

Tasks

Part-Of-Speech Tagging Transfer Learning

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Residual Connection 설명 없음

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

Weight Decay 설명 없음

Refunds@Expedia|||How do I get a full refund from Expedia? “How do I get a full refund from Expedia? How do I get a full refund from Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Quick Help &…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Adam 설명 없음

What Does BERT Learn about the Structure of Language?

2019-07-01 · ACL 2019 7 · Ganesh Jawahar, Beno{\^\i}t Sagot, Djam{\'e} Seddah

BERT is a recent language representation model that has surprisingly performed well in diverse language understanding benchmarks. This result indicates the possibility that BERT networks capture structural information ab…

What do Transformers Know about Government?

2024-04-22 · Jue Hou, Anisia Katinskaia, Lari Kotilainen, Sathianpong Trangcasanchai 외

This paper investigates what insights about linguistic features and what knowledge about the structure of natural language can be obtained from the encodings in transformer language models.In particular, we explore how B…

Sentence

What does BERT learn about prosody?

2023-04-25 · Sofoklis Kakouros, Johannah O'Mahony

Language models have become nearly ubiquitous in natural language processing applications achieving state-of-the-art results in many tasks including prosody. As the model design does not define predetermined linguistic t…

TinyBERT: Distilling BERT for Natural Language Understanding

2019-09-23 · Findings of the Association for Computational Linguistics 2020 · Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang 외

Language model pre-training, such as BERT, has significantly improved the performances of many natural language processing tasks. However, pre-trained language models are usually computationally expensive, so it is diffi…

Knowledge Distillation Language Modelling Linguistic Acceptability Natural Language Inference+5

INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers

2023-12-08 · Anjan Karmakar, Romain Robbes

Pre-trained models of source code have recently been successfully applied to a wide variety of Software Engineering tasks; they have also seen some practical adoption in practice, e.g. for code completion. Yet, we still …

Code Completion Diagnostic Language Modelling