A Comparative Analysis of Word Segmentation, Part-of-Speech Tagging, and Named Entity Recognition for Historical Chinese Sources, 1900-1950
This paper compares large language models (LLMs) and traditional natural language processing (NLP) tools for performing word segmentation, part-of-speech (POS) tagging, and named entity recognition (NER) on Chinese texts from 1900 to 1950. Historical Chinese documents pose challenges for text analysis due to their logographic script, the absence of natural word boundaries, and significant linguistic changes. Using a sample dataset from the Shanghai Library Republican Journal corpus, traditional tools such as Jieba and spaCy are compared to LLMs, including GPT-4o, Claude 3.5, and the GLM series. The results show that LLMs outperform traditional methods in all metrics, albeit at considerably higher computational costs, highlighting a trade-off between accuracy and efficiency. Additionally, LLMs better handle genre-specific challenges such as poetry and temporal variations (i.e., pre-1920 versus post-1920 texts), demonstrating that their contextual learning capabilities can advance NLP approaches to historical texts by reducing the need for domain-specific training data.
Code (0)
등록된 구현이 없습니다.
Tasks
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)NERPart-Of-Speech TaggingPOSPOS TaggingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Automatic Word Segmentation and Part-of-Speech Tagging of Ancient Chinese Based on BERT Model
In recent years, new deep learning methods and pre-training language models have been emerging in the field of natural language processing (NLP). These methods and models can greatly improve the accuracy of automatic wor…
Language ModelingLanguage ModellingMachine Reading ComprehensionPart-Of-Speech Tagging+4What Do Self-Supervised Speech Models Know About Words?
Many self-supervised speech models (S3Ms) have been introduced over the last few years, improving performance and data efficiency on various speech tasks. However, these empirical successes alone do not give a complete p…
SentenceSentence SimilarityVisual GroundingIntegration of Automatic Sentence Segmentation and Lexical Analysis of Ancient Chinese based on BiLSTM-CRF Model
The basic tasks of ancient Chinese information processing include automatic sentence segmentation, word segmentation, part-of-speech tagging and named entity recognition. Tasks such as lexical analysis need to be based o…
Lexical Analysisnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)+4Phonetic Segmentation of the UCLA Phonetics Lab Archive
Research in speech technologies and comparative linguistics depends on access to diverse and accessible speech data. The UCLA Phonetics Lab Archive is one of the earliest multilingual speech corpora, with long-form audio…
BERT 4EVER@EvaHan 2022: Ancient Chinese Word Segmentation and Part-of-Speech Tagging Based on Adversarial Learning and Continual Pre-training
With the development of artificial intelligence (AI) and digital humanities, ancient Chinese resources and language technology have also developed and grown, which have become an increasingly important part to the study …
Chinese Word SegmentationCultural Vocal Bursts Intensity PredictionEnsemble LearningPart-Of-Speech Tagging+3