BioM-Transformers: Building Large Biomedical Language Models with BERT, ALBERT and ELECTRA
The impact of design choices on the performance of biomedical language models recently has been a subject for investigation. In this paper, we empirically study biomedical domain adaptation with large transformer models using different design choices. We evaluate the performance of our pretrained models against other existing biomedical language models in the literature. Our results show that we achieve state-of-the-art results on several biomedical domain tasks despite using similar or less computational cost compared to other models in the literature. Our findings highlight the significant effect of design choices on improving the performance of biomedical language models.
Code (1)
Tasks
Domain AdaptationRelation ExtractionSimilar Papers 제목 키워드 기반
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Biomedical text mining is becoming increasingly important as the number of biomedical documents rapidly grows. With the progress in natural language processing (NLP), extracting valuable information from biomedical liter…
Drug–drug Interaction ExtractionFew-Shot LearningLanguage ModellingMedical Named Entity Recognition+8On the Effectiveness of Compact Biomedical Transformers
Language models pre-trained on biomedical corpora, such as BioBERT, have recently shown promising results on downstream biomedical tasks. Many existing pre-trained models, on the other hand, are resource-intensive and co…
Continual LearningKnowledge DistillationLanguage ModellingNamed Entity Recognition (NER)Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformers
ChatGPT is a large language model developed by OpenAI. Despite its impressive performance across various tasks, no prior work has investigated its capability in the biomedical domain yet. To this end, this paper aims to …
Document ClassificationLanguage ModelingLanguage ModellingLarge Language Model+2Comparing Encoder-Only and Encoder-Decoder Transformers for Relation Extraction from Biomedical Texts: An Empirical Study on Ten Benchmark Datasets
Biomedical relation extraction, aiming to automatically discover high-quality and semantic relations between the entities from free text, is becoming a vital step for automated knowledge discovery. Pretrained language mo…
DecoderMulti-Task LearningRelationRelation ExtractionBuilding a Biomedical Full-Text Part-of-Speech Corpus Semi-Automatically
This paper presents a method for semi-automatically building a corpus of full-text English-language biomedical articles annotated with part-of-speech tags. The outcomes are a semi-automatic procedure to create a large si…
ArticlesTAG