Context Analysis for Pre-trained Masked Language Models

2020-11-01 · Findings of the Association for Computational Linguistics 2020 · Yi-An Lai, Garima Lalwani, Yi Zhang

Pre-trained language models that learn contextualized word representations from a large un-annotated corpus have become a standard component for many state-of-the-art NLP systems. Despite their successful applications in various downstream NLP tasks, the extent of contextual impact on the word representation has not been explored. In this paper, we present a detailed analysis of contextual impact in Transformer- and BiLSTM-based masked language models. We follow two different approaches to evaluate the impact of context: a masking based approach that is architecture agnostic, and a gradient based approach that requires back-propagation through networks. The findings suggest significant differences on the contextual impact between the two model architectures. Through further breakdown of analysis by syntactic categories, we find the contextual impact in Transformer-based MLM aligns well with linguistic intuition. We further explore the Transformer attention pruning based on our findings in contextual analysis.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Pruning 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Attention 설명 없음

A Unified Framework for In-Context Learning with Causal and Masked Language Models

2026-07-05 · Chenrui Liu, Chuanlong Xie, Falong Tan, Yicheng Zeng 외 arxiv

In-context learning (ICL) has emerged as a central capability of pretrained language models, yet its theoretical analysis has focused primarily on causal language models trained by left-to-right autoregressive prediction…

MeLT: Message-Level Transformer with Masked Document Representations as Pre-Training for Stance Detection

2021-09-16 · Findings (EMNLP) 2021 11 · Matthew Matero, Nikita Soni, Niranjan Balasubramanian, H. Andrew Schwartz

Much of natural language processing is focused on leveraging large capacity language models, typically trained over single messages with a task of predicting one or more tokens. However, modeling human language at higher…

Attribute Language Modeling Language Modelling Masked Language Modeling+1

Conditional BERT Contextual Augmentation

2018-12-17 · Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han 외

We propose a novel data augmentation method for labeled sentences called conditional BERT contextual augmentation. Data augmentation methods are often applied to prevent overfitting and improve generalization of deep neu…

Data Augmentation Language Modeling Language Modelling Text Classification

Contextual Distortion Reveals Constituency: Masked Language Models are Implicit Parsers

2023-06-01 · Jiaxi Li, Wei Lu

Recent advancements in pre-trained language models (PLMs) have demonstrated that these models possess some degree of syntactic awareness. To leverage this knowledge, we propose a novel chart-based method for extracting p…

DICT-MLM: Improved Multilingual Pre-Training using Bilingual Dictionaries

2020-10-23 · Aditi Chaudhary, Karthik Raman, Krishna Srinivasan, Jiecao Chen

Pre-trained multilingual language models such as mBERT have shown immense gains for several natural language processing (NLP) tasks, especially in the zero-shot cross-lingual setting. Most, if not all, of these pre-train…

Language Modeling Language Modelling Masked Language Modeling Representation Learning