SciBERT: A Pretrained Language Model for Scientific Text
Obtaining large-scale annotated data for NLP tasks in the scientific domain is challenging and expensive. We release SciBERT, a pretrained language model based on BERT (Devlin et al., 2018) to address the lack of high-quality, large-scale labeled scientific data. SciBERT leverages unsupervised pretraining on a large multi-domain corpus of scientific publications to improve performance on downstream scientific NLP tasks. We evaluate on a suite of tasks including sequence tagging, sentence classification and dependency parsing, with datasets from a variety of scientific domains. We demonstrate statistically significant improvements over BERT and achieve new state-of-the-art results on several of these tasks. The code and pretrained models are available at https://github.com/allenai/scibert/.
Code (6)
Tasks
Citation Intent ClassificationDependency ParsingGeneral ClassificationLanguage ModelingLanguage ModellingMedical Named Entity RecognitionmodelNamed Entity Recognition (NER)Participant Intervention Comparison Outcome ExtractionRelation ExtractionSentenceSentence ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction
An overwhelmingly large amount of knowledge in the materials domain is generated and stored as text published in peer-reviewed scientific literature. Recent developments in natural language processing, such as bidirectio…
Language ModelingLanguage Modellingnamed-entity-recognitionNamed Entity Recognition+2COVID-19 therapy target discovery with context-aware literature mining
The abundance of literature related to the widespread COVID-19 pandemic is beyond manual inspection of a single expert. Development of systems, capable of automatically processing tens of thousands of scientific publicat…
Domain AdaptationLanguage ModelingLanguage ModellingLiterature Mining+1Fine-Tuning Large Language Models for Scientific Text Classification: A Comparative Study
The exponential growth of online textual content across diverse domains has necessitated advanced methods for automated text classification. Large Language Models (LLMs) based on transformer architectures have shown sign…
Classificationtext-classificationText ClassificationScientific Keyphrase Identification and Classification by Pre-Trained Language Models Intermediate Task Transfer Learning
Scientific keyphrase identification and classification is the task of detecting and classifying keyphrases from scholarly text with their types from a set of predefined classes. This task has a wide range of benefits, bu…
ClassificationPOSPOS TaggingTransfer LearningLeveraging Domain Agnostic and Specific Knowledge for Acronym Disambiguation
An obstacle to scientific document understanding is the extensive use of acronyms which are shortened forms of long technical phrases. Acronym disambiguation aims to find the correct meaning of an ambiguous acronym in a …
document understandingWord Embeddings