BERT's output layer recognizes all hidden layers? Some Intriguing Phenomena and a simple way to boost BERT
Although Bidirectional Encoder Representations from Transformers (BERT) have achieved tremendous success in many natural language processing (NLP) tasks, it remains a black box. A variety of previous works have tried to lift the veil of BERT and understand each layer's functionality. In this paper, we found that surprisingly the output layer of BERT can reconstruct the input sentence by directly taking each layer of BERT as input, even though the output layer has never seen the input other than the final hidden layer. This fact remains true across a wide variety of BERT-based models, even when some layers are duplicated. Based on this observation, we propose a quite simple method to boost the performance of BERT. By duplicating some layers in the BERT-based models to make it deeper (no extra training required in this step), they obtain better performance in the downstream tasks after fine-tuning.
Code (0)
등록된 구현이 없습니다.
Tasks
AllSentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Jump to Conclusions: Short-Cutting Transformers With Linear Transformations
Transformer-based language models create hidden representations of their inputs at every layer, but only use final-layer representations for prediction. This obscures the internal decision-making process of the model and…
Decision MakingLanguage ModelingLanguage ModellingGRUBERT: A GRU-Based Method to Fuse BERT Hidden Layers for Twitter Sentiment Analysis
In this work, we introduce a GRU-based architecture called GRUBERT that learns to map the different BERT hidden layers to fused embeddings with the aim of achieving high accuracy on the Twitter sentiment analysis task. T…
Sentiment AnalysisTwitter Sentiment AnalysisUniversal-KD: Attention-based Output-Grounded Intermediate Layer Knowledge Distillation
Intermediate layer matching is shown as an effective approach for improving knowledge distillation (KD). However, this technique applies matching in the hidden spaces of two different networks (i.e. student and teacher),…
Knowledge DistillationPatient Knowledge Distillation for BERT Model Compression
Pre-trained language models such as BERT have proven to be highly effective for natural language processing (NLP) tasks. However, the high demand for computing resources in training such models hinders their application …
Knowledge DistillationmodelModel CompressionBreaking Down Word Semantics from Pre-trained Language Models through Layer-wise Dimension Selection
Contextual word embeddings obtained from pre-trained language model (PLM) have proven effective for various natural language processing tasks at the word level. However, interpreting the hidden aspects within embeddings,…
Binary ClassificationLanguage ModelingLanguage ModellingRepresentation Learning+1