NukeLM: Pre-Trained and Fine-Tuned Language Models for the Nuclear and Energy Domains
Natural language processing (NLP) tasks (text classification, named entity recognition, etc.) have seen revolutionary improvements over the last few years. This is due to language models such as BERT that achieve deep knowledge transfer by using a large pre-trained model, then fine-tuning the model on specific tasks. The BERT architecture has shown even better performance on domain-specific tasks when the model is pre-trained using domain-relevant texts. Inspired by these recent advancements, we have developed NukeLM, a nuclear-domain language model pre-trained on 1.5 million abstracts from the U.S. Department of Energy Office of Scientific and Technical Information (OSTI) database. This NukeLM model is then fine-tuned for the classification of research articles into either binary classes (related to the nuclear fuel cycle [NFC] or not) or multiple categories related to the subject of the article. We show that continued pre-training of a BERT-style architecture prior to fine-tuning yields greater performance on both article classification tasks. This information is critical for properly triaging manuscripts, a necessary task for better understanding citation networks that publish in the nuclear space, and for uncovering new areas of research in the nuclear (or nuclear-relevant) domains.
Code (0)
등록된 구현이 없습니다.
Tasks
ArticlesLanguage Modellingnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)text-classificationText ClassificationTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Electron-nucleus cross sections from transfer learning
Transfer learning (TL) allows a deep neural network (DNN) trained on one type of data to be adapted for new problems with limited information. We propose to use the TL technique in physics. The DNN learns the physics of …
Transfer LearningMechanistic Interpretability of LoRA-Adapted Language Models for Nuclear Reactor Safety Applications
The integration of Large Language Models (LLMs) into safety-critical domains, such as nuclear engineering, necessitates a deep understanding of their internal reasoning processes. This paper presents a novel methodology …
parameter-efficient fine-tuningQuantified limits of the nuclear landscape
The chart of the nuclides is limited by particle drip lines beyond which nuclear stability to proton or neutron emission is lost. Predicting the range of particle-bound isotopes poses an appreciable challenge for nuclear…
Gaussian ProcessesTowards Secure and Private Language Models for Nuclear Power Plants
This paper introduces a domain-specific Large Language Model for nuclear applications, built from the publicly accessible Essential CANDU textbook. Drawing on a compact Transformer-based architecture, the model is traine…
GPULanguage ModelingLanguage ModellingLarge Language Model+1NukeBERT: A Pre-trained language model for Low Resource Nuclear Domain
Significant advances have been made in recent years on Natural Language Processing with machines surpassing human performance in many tasks, including but not limited to Question Answering. The majority of deep learning …
Language ModelingLanguage ModellingQuestion Answering