Small and Practical BERT Models for Sequence Labeling
We propose a practical scheme to train a single multilingual sequence labeling model that yields state of the art results and is small and fast enough to run on a single CPU. Starting from a public multilingual BERT checkpoint, our final model is 6x smaller and 27x faster, and has higher accuracy than a state-of-the-art multilingual baseline. We show that our model especially outperforms on low-resource languages, and works on codemixed input text without being explicitly trained on codemixed examples. We showcase the effectiveness of our method by reporting on part-of-speech tagging and morphological prediction on 70 treebanks and 48 languages.
Code (0)
등록된 구현이 없습니다.
Tasks
CPUPart-Of-Speech TaggingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Training sequence labeling models using prior knowledge
Sequence labeling task (part-of-speech tagging, named entity recognition) is one of the most common in NLP. At different times, the following architectures were used to solve it: CRF, BiLSTM, BERT (in chronological order…
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)Part-Of-Speech TaggingWhich Student is Best? A Comprehensive Knowledge Distillation Exam for Task-Specific BERT Models
We perform knowledge distillation (KD) benchmark from task-specific BERT-base teacher models to various student models: BiLSTM, CNN, BERT-Tiny, BERT-Mini, and BERT-Small. Our experiment involves 12 datasets grouped in tw…
CPUData AugmentationKnowledge Distillationtext-classification+2BERTatDE at SemEval-2020 Task 6: Extracting Term-definition Pairs in Free Text Using Pre-trained Model
Definition extraction is an important task in Nature Language Processing, and it is used to identify the terms and definitions related to terms. The task contains sentence classification task (i.e., classify whether it c…
Definition ExtractionSentenceSentence ClassificationCLaC at SemEval-2023 Task 2: Comparing Span-Prediction and Sequence-Labeling approaches for NER
This paper summarizes the CLaC submission for the MultiCoNER 2 task which concerns the recognition of complex, fine-grained named entities. We compare two popular approaches for NER, namely Sequence Labeling and Span Pre…
NERPredictionTask 2Accelerating BERT Inference for Sequence Labeling via Early-Exit
Both performance and efficiency are crucial factors for sequence labeling tasks in many real-world scenarios. Although the pre-trained models (PTMs) have significantly improved the performance of various sequence labelin…
Sentence