Exploring the Benefits of Domain-Pretraining of Generative Large Language Models for Chemistry
A proliferation of Large Language Models (the GPT series, BLOOM, LLaMA, and more) are driving forward novel development of multipurpose AI for a variety of tasks, particularly natural language processing (NLP) tasks. These models demonstrate strong performance on a range of tasks; however, there has been evidence of brittleness when applied to more niche or narrow domains where hallucinations or fluent but incorrect responses reduce performance. Given the complex nature of scientific domains, it is prudent to investigate the trade-offs of leveraging off-the-shelf versus more targeted foundation models for scientific domains. In this work, we examine the benefits of in-domain pre-training for a given scientific domain, chemistry, and compare these to open-source, off-the-shelf models with zero-shot and few-shot prompting. Our results show that not only do in-domain base models perform reasonably well on in-domain tasks in a zero-shot setting but that further adaptation using instruction fine-tuning yields impressive performance on chemistry-specific tasks such as named entity recognition and molecular formula generation.
Code (0)
등록된 구현이 없습니다.
Tasks
named-entity-recognitionNamed Entity RecognitionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Exploring Limits of Diffusion-Synthetic Training with Weakly Supervised Semantic Segmentation
The advance of generative models for images has inspired various training techniques for image recognition utilizing synthetic images. In semantic segmentation, one promising approach is extracting pseudo-masks from atte…
Data AugmentationImage GenerationSegmentationSemantic Segmentation+4Video Pretraining Advances 3D Deep Learning on Chest CT Tasks
Pretraining on large natural image classification datasets such as ImageNet has aided model development on data-scarce 2D medical tasks. 3D medical tasks often have much less data than 2D medical tasks, prompting practit…
Deep Learningimage-classificationImage ClassificationGuiding Pretraining in Reinforcement Learning with Large Language Models
Reinforcement learning algorithms typically struggle in the absence of a dense, well-shaped reward function. Intrinsically motivated exploration methods address this limitation by rewarding agents for visiting novel stat…
Common Sense ReasoningLanguage ModelingLanguage Modellingreinforcement-learning+2MDAPT: Multilingual Domain Adaptive Pretraining in a Single Model
Domain adaptive pretraining, i.e. the continued unsupervised pretraining of a language model on domain-specific text, improves the modelling of text for downstream tasks within the domain. Numerous real-world application…
Language ModelingLanguage Modellingnamed-entity-recognitionNamed Entity Recognition+3Precision at Scale: Domain-Specific Datasets On-Demand
In the realm of self-supervised learning (SSL), conventional wisdom has gravitated towards the utility of massive, general domain datasets for pretraining robust backbones. In this paper, we challenge this idea by explor…
Self-Supervised Learning