An Empirical Study of Multi-Task Learning on BERT for Biomedical Text Mining
Multi-task learning (MTL) has achieved remarkable success in natural language processing applications. In this work, we study a multi-task learning model with multiple decoders on varieties of biomedical and clinical natural language processing tasks such as text similarity, relation extraction, named entity recognition, and text inference. Our empirical results demonstrate that the MTL fine-tuned models outperform state-of-the-art transformer models (e.g., BERT and its variants) by 2.0% and 1.3% in biomedical and clinical domains, respectively. Pairwise MTL further demonstrates more details about which tasks can improve or decrease others. This is particularly helpful in the context that researchers are in the hassle of choosing a suitable model for new problems. The code and models are publicly available at https://github.com/ncbi-nlp/bluebert
Code (1)
Tasks
Multi-Task Learningnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)Relation Extractiontext similarityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Comparing Encoder-Only and Encoder-Decoder Transformers for Relation Extraction from Biomedical Texts: An Empirical Study on Ten Benchmark Datasets
Biomedical relation extraction, aiming to automatically discover high-quality and semantic relations between the entities from free text, is becoming a vital step for automated knowledge discovery. Pretrained language mo…
DecoderMulti-Task LearningRelationRelation ExtractionBioM-Transformers: Building Large Biomedical Language Models with BERT, ALBERT and ELECTRA
The impact of design choices on the performance of biomedical language models recently has been a subject for investigation. In this paper, we empirically study biomedical domain adaptation with large transformer models …
Domain AdaptationRelation ExtractionAdaptation of Biomedical and Clinical Pretrained Models to French Long Documents: A Comparative Study
Recently, pretrained language models based on BERT have been introduced for the French biomedical domain. Although these models have achieved state-of-the-art results on biomedical and clinical NLP tasks, they are constr…
named-entity-recognitionNamed Entity RecognitionBERT-based Ranking for Biomedical Entity Normalization
Developing high-performance entity normalization algorithms that can alleviate the term variation problem is of great interest to the biomedical community. Although deep learning-based methods have been successfully appl…
Word EmbeddingsBioBERT: a pre-trained biomedical language representation model for biomedical text mining
Biomedical text mining is becoming increasingly important as the number of biomedical documents rapidly grows. With the progress in natural language processing (NLP), extracting valuable information from biomedical liter…
Drug–drug Interaction ExtractionFew-Shot LearningLanguage ModellingMedical Named Entity Recognition+8