BERT4GCN: Using BERT Intermediate Layers to Augment GCN for Aspect-based Sentiment Classification

2021-10-01 · EMNLP 2021 11 · Zeguan Xiao, Jiarun Wu, Qingliang Chen, Congjian Deng

Graph-based Aspect-based Sentiment Classification (ABSC) approaches have yielded state-of-the-art results, expecially when equipped with contextual word embedding from pre-training language models (PLMs). However, they ignore sequential features of the context and have not yet made the best of PLMs. In this paper, we propose a novel model, BERT4GCN, which integrates the grammatical sequential features from the PLM of BERT, and the syntactic knowledge from dependency graphs. BERT4GCN utilizes outputs from intermediate layers of BERT and positional information between words to augment GCN (Graph Convolutional Network) to better encode the dependency graphs for the downstream classification. Experimental results demonstrate that the proposed BERT4GCN outperforms all state-of-the-art baselines, justifying that augmenting GCN with the grammatical features from intermediate layers of BERT can significantly empower ABSC models.

📄 PDF Abstract BibTeX arXiv:2110.00171

Code (0)

등록된 구현이 없습니다.

Tasks

Sentiment AnalysisSentiment Classification

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

WordPiece 설명 없음

Adam 설명 없음

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

Residual Connection 설명 없음

Weight Decay 설명 없음

Utilizing BERT Intermediate Layers for Aspect Based Sentiment Analysis and Natural Language Inference

2020-02-12 · Youwei Song, Jiahai Wang, Zhiwei Liang, Zhiyue Liu 외

Aspect based sentiment analysis aims to identify the sentimental tendency towards a given aspect in text. Fine-tuning of pretrained BERT performs excellent on this task and achieves state-of-the-art performances. Existin…

Aspect-Based Sentiment Analysis Aspect-Based Sentiment Analysis (ABSA)Natural Language Inference Sentiment Analysis

BERT-JAM: Boosting BERT-Enhanced Neural Machine Translation with Joint Attention

2020-11-09 · Zhebin Zhang, Sai Wu, Dawei Jiang, Gang Chen

BERT-enhanced neural machine translation (NMT) aims at leveraging BERT-encoded representations for translation tasks. A recently proposed approach uses attention mechanisms to fuse Transformer's encoder and decoder layer…

Decoder Machine Translation NMT Translation

Undivided Attention: Are Intermediate Layers Necessary for BERT?

2020-12-22 · Sharath Nittur Sridhar, Anthony Sarah

In recent times, BERT-based models have been extremely successful in solving a variety of natural language processing (NLP) tasks such as reading comprehension, natural language inference, sentiment analysis, etc. All BE…

Natural Language Inference Reading Comprehension Sentiment Analysis

CEEBERT: Cross-Domain Inference in Early Exit BERT

2024-05-23 · Divya Jyoti Bajpai, Manjesh Kumar Hanawal

Pre-trained Language Models (PLMs), like BERT, with self-supervision objectives exhibit remarkable performance and generalization across various tasks. However, they suffer in inference latency due to their large size. T…

Domain Adaptation

BERTnesia: Investigating the capture and forgetting of knowledge in BERT

2020-10-19 · Jonas Wallat, Jaspreet Singh, Avishek Anand

Probing complex language models has recently revealed several insights into linguistic and semantic patterns found in the learned representations. In this paper, we probe BERT specifically to understand and measure the r…

Knowledge Base Completion NER Question Answering