Astro-NER -- Astronomy Named Entity Recognition: Is GPT a Good Domain Expert Annotator?
In this study, we address one of the challenges of developing NER models for scholarly domains, namely the scarcity of suitable labeled data. We experiment with an approach using predictions from a fine-tuned LLM model to aid non-domain experts in annotating scientific entities within astronomy literature, with the goal of uncovering whether such a collaborative process can approximate domain expertise. Our results reveal moderate agreement between a domain expert and the LLM-assisted non-experts, as well as fair agreement between the domain expert and the LLM model's predictions. In an additional experiment, we compare the performance of finetuned and default LLMs on this task. We have also introduced a specialized scientific entity annotation scheme for astronomy, validated by a domain expert. Our approach adopts a scholarly research contribution-centric perspective, focusing exclusively on scientific entities relevant to the research theme. The resultant dataset, containing 5,000 annotated astronomy article titles, is made publicly available.
Code (0)
등록된 구현이 없습니다.
Tasks
Astronomynamed-entity-recognitionNamed Entity RecognitionNERSimilar Papers 제목 키워드 기반
Building astroBERT, a language model for Astronomy & Astrophysics
The existing search tools for exploring the NASA Astrophysics Data System (ADS) can be quite rich and empowering (e.g., similar and trending operators), but researchers are not yet allowed to fully leverage semantic sear…
AstronomyLanguage ModelingLanguage Modellingmodel+3Identifying Planetary Names in Astronomy Papers: A Multi-Step Approach
The automatic identification of planetary feature names in astronomy publications presents numerous challenges. These features include craters, defined as roughly circular depressions resulting from impact or volcanic ac…
AstronomyLanguage ModellingLarge Language Modelnamed-entity-recognition+6ASTROMLSKIT: A New Statistical Machine Learning Toolkit: A Platform for Data Analytics in Astronomy
Astroinformatics is a new impact area in the world of astronomy, occasionally called the final frontier, where several astrophysicists, statisticians and computer scientists work together to tackle various data intensive…
AstronomyBIG-bench Machine LearningGeneral ClassificationContinual Named Entity Recognition without Catastrophic Forgetting
Continual Named Entity Recognition (CNER) is a burgeoning area, which involves updating an existing model by incorporating new entity types sequentially. Nevertheless, continual learning approaches are often severely aff…
Continual LearningContinual Named Entity Recognitionnamed-entity-recognitionNamed Entity RecognitionDesigning an Evaluation Framework for Large Language Models in Astronomy Research
Large Language Models (LLMs) are shifting how scientific research is done. It is imperative to understand how researchers interact with these models and how scientific sub-communities like astronomy might benefit from th…
AstronomyChatbotExperimental DesignRAG+2