On Adversarial Examples for Biomedical NLP Tasks
The success of pre-trained word embeddings has motivated its use in tasks in the biomedical domain. The BERT language model has shown remarkable results on standard performance metrics in tasks such as Named Entity Recognition (NER) and Semantic Textual Similarity (STS), which has brought significant progress in the field of NLP. However, it is unclear whether these systems work seemingly well in critical domains, such as legal or medical. For that reason, in this work, we propose an adversarial evaluation scheme on two well-known datasets for medical NER and STS. We propose two types of attacks inspired by natural spelling errors and typos made by humans. We also propose another type of attack that uses synonyms of medical terms. Under these adversarial settings, the accuracy of the models drops significantly, and we quantify the extent of this performance loss. We also show that we can significantly improve the robustness of the models by training them with adversarial examples. We hope our work will motivate the use of adversarial examples to evaluate and develop models with increased robustness for medical tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage Modellingnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)NERSemantic Textual SimilaritySTSWord EmbeddingsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Adversarial Evaluation of BERT for Biomedical Named Entity Recognition
The success of pre-trained word embeddings of the BERT model has motivated its use in tasks in the biomedical domain. However, it is not clear if this model works correctly in real scenarios. In this work, we propose an …
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)Word EmbeddingsDetecting Adversarial Examples in Batches -- a geometrical approach
Many deep learning methods have successfully solved complex tasks in computer vision and speech recognition applications. Nonetheless, the robustness of these models has been found to be vulnerable to perturbed inputs or…
Adversarial AttackAdversarial Attack DetectionImage ClassificationStress Test Evaluation of Biomedical Word Embeddings
The success of pretrained word embeddings has motivated their use in the biomedical domain, with contextualized embeddings yielding remarkable results in several biomedical NLP tasks. However, there is a lack of research…
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)NER+1Impact of Adversarial Examples on Deep Learning Models for Biomedical Image Segmentation
Deep learning models, which are increasingly being used in the field of medical image analysis, come with a major security risk, namely, their vulnerability to adversarial examples. Adversarial examples are carefully cra…
Image SegmentationLesion SegmentationMedical Image AnalysisOptic Disc Segmentation+3Assessing biomedical knowledge robustness in large language models by query-efficient sampling attacks
The increasing depth of parametric domain knowledge in large language models (LLMs) is fueling their rapid deployment in real-world applications. Understanding model vulnerabilities in high-stakes and knowledge-intensive…
Distractor GenerationQuestion Answering