Does Chinese BERT Encode Word Structure?
Contextualized representations give significantly improved results for a wide range of NLP tasks. Much work has been dedicated to analyzing the features captured by representative models such as BERT. Existing work finds that syntactic, semantic and word sense knowledge are encoded in BERT. However, little work has investigated word features for character-based languages such as Chinese. We investigate Chinese BERT using both attention weight distribution statistics and probing tasks, finding that (1) word information is captured by BERT; (2) word-level features are mostly in the middle representation layers; (3) downstream tasks make different use of word features in BERT, with POS tagging and chunking relying the most on word features, and natural language inference relying the least on such features.
Code (1)
Tasks
ChunkingNatural Language InferencePOSPOS TaggingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations
The pre-training of text encoders normally processes text as a sequence of tokens corresponding to small text units, such as word pieces in English and characters in Chinese. It omits information carried by larger text g…
Chinese Named Entity RecognitionChinese Word SegmentationDocument ClassificationNatural Language Inference+3Lex-BERT: Enhancing BERT based NER with lexicons
In this work, we represent Lex-BERT, which incorporates the lexicon information into Chinese BERT for named entity recognition (NER) tasks in a natural manner. Instead of using word embeddings and a newly designed transf…
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)NER+2A Reproducible Multi-Architecture Baseline for Token-Level Chinese Metaphor Identification under the MIPVU Framework
Metaphor is pervasive in everyday language, yet token-level computational identification of metaphor-related words in Chinese under the MIPVU framework remains under-explored relative to English. This paper presents a re…
MVP-BERT: Redesigning Vocabularies for Chinese BERT and Multi-Vocab Pretraining
Despite the development of pre-trained language models (PLMs) significantly raise the performances of various Chinese natural language processing (NLP) tasks, the vocabulary for these Chinese PLMs remain to be the one pr…
Chinese Word SegmentationLanguage ModelingLanguage ModellingSentenceAMBERT: A Pre-trained Language Model with Multi-Grained Tokenization
Pre-trained language models such as BERT have exhibited remarkable performances in many tasks in natural language understanding (NLU). The tokens in the models are usually fine-grained in the sense that for languages lik…
Language ModelingLanguage ModellingNatural Language Understanding