Diacritics Restoration using BERT with Analysis on Czech language

2021-05-24 · Jakub Náplava, Milan Straka, Jana Straková

We propose a new architecture for diacritics restoration based on contextualized embeddings, namely BERT, and we evaluate it on 12 languages with diacritics. Furthermore, we conduct a detailed error analysis on Czech, a morphologically rich language with a high level of diacritization. Notably, we manually annotate all mispredictions, showing that roughly 44% of them are actually not errors, but either plausible variants (19%), or the system corrections of erroneous data (25%). Finally, we categorize the real errors in detail. We release the code at https://github.com/ufal/bert-diacritics-restoration.

📄 PDF Abstract BibTeX arXiv:2105.11408

Code (1)

ufal/bert-diacritics-restoration 공식 구현 pytorch

Tasks

Croatian Text Diacritization Czech Text Diacritization French Text Diacritization Hungarian Text Diacritization Irish Text Diacritization Latvian Text Diacritization Polish Text Diacritization Romanian Text Diacritization Slovak Text Diacritization Spanish Text Diacritization Turkish Text Diacritization Vietnamese Text Diacritization

Methods 이 논문이 사용한 방법론

Multi-Head Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

WordPiece 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Refunds@Expedia|||How do I get a full refund from Expedia? “How do I get a full refund from Expedia? How do I get a full refund from Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Quick Help &…

Romanian Diacritics Restoration Using Recurrent Neural Networks

2020-09-06 · Stefan Ruseti, Teodor-Mihai Cotet, Mihai Dascalu

Diacritics restoration is a mandatory step for adequately processing Romanian texts, and not a trivial one, as you generally need context in order to properly restore a character. Most previous methods which were experim…

RoBERT -- A Romanian BERT Model

2020-12-01 · COLING 2020 8 · Mihai Masala, Stefan Ruseti, Mihai Dascalu

Deep pre-trained language models tend to become ubiquitous in the field of Natural Language Processing (NLP). These models learn contextualized representations by using a huge amount of unlabeled text data and obtain sta…

model Sentiment Analysis Transfer Learning

Lexical Disambiguation of Igbo using Diacritic Restoration

2017-04-01 · WS 2017 4 · Ignatius Ezeani, Mark Hepple, Ikechukwu Onyenwe

Properly written texts in Igbo, a low-resource African language, are rich in both orthographic and tonal diacritics. Diacritics are essential in capturing the distinctions in pronunciation and meaning of words, as well a…

BIG-bench Machine Learning General Classification

The Degree of Language Diacriticity and Its Effect on Tasks

2026-03-29 · Adi Cohen, Yuval Pinter arxiv

Diacritics are orthographic marks that clarify pronunciation, distinguish similar words, or alter meaning. They play a central role in many writing systems, yet their impact on language technology has not been systematic…

Correcting diacritics and typos with a ByT5 transformer model

2022-01-31 · Lukas Stankevičius, Mantas Lukoševičius, Jurgita Kapočiūtė-Dzikienė, Monika Briedienė 외

Due to the fast pace of life and online communications and the prevalence of English and the QWERTY keyboard, people tend to forgo using diacritics, make typographical errors (typos) when typing in other languages. Resto…