Punctuation restoration Model and Spacing Model for Korean Ancient Document
In Korean ancient documents, there is no spacing or punctuation, and they are written in classical Chinese characters. This makes it challenging for modern individuals and translation models to accurately interpret and translate them. While China has models predicting punctuation and spacing, applying them directly to Korean texts is problematic due to data differences. Therefore, we developed the first models which predict punctuation and spacing for Korean historical texts and evaluated their performance. Our punctuation restoration model achieved an F1 score of 0.84, and Spacing model achieved a score of 0.96. It has the advantage of enabling inference on low-performance GPUs with less VRAM while maintaining quite high accuracy.
Code (0)
등록된 구현이 없습니다.
Tasks
modelPunctuation RestorationTranslationSimilar Papers 제목 키워드 기반
When Does Classical Chinese Help? Quantifying Cross-Lingual Transfer in Hanja and Kanbun
Historical and linguistic connections within the Sinosphere have led researchers to use Classical Chinese resources for cross-lingual transfer when processing historical documents from Korea and Japan. In this paper, we …
Cross-Lingual TransferLanguage ModelingLanguage ModellingMachine Translation+4HERITAGE: An End-to-End Web Platform for Processing Korean Historical Documents in Hanja
While Korean historical documents are invaluable cultural heritage, understanding those documents requires in-depth Hanja expertise. Hanja is an ancient language used in Korea before the 20th century, whose characters we…
document understandingMachine Translationnamed-entity-recognitionNamed Entity Recognition+1Priming Ancient Korean Neural Machine Translation
In recent years, there has been an increasing need for the restoration and translation of historical languages. In this study, we attempt to translate historical records in ancient Korean language based on neural machine…
Machine TranslationNMTTranslationBoosting Punctuation Restoration with Data Generation and Reinforcement Learning
Punctuation restoration is an important task in automatic speech recognition (ASR) which aim to restore the syntactic structure of generated ASR texts to improve readability. While punctuated texts are abundant from writ…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Punctuation Restorationreinforcement-learning+3Predicting Punctuation in Ancient Chinese Texts: A Multi-Layered LSTM and Attention-Based Approach
It was only until the 20th century when the Chinese language began using punctuation. In fact, many ancient Chinese texts contain thousands of lines with no distinct punctuation marks or delimiters in sight. The lack of …