Contextual Text Denoising with Masked Language Models
Recently, with the help of deep learning models, significant advances have been made in different Natural Language Processing (NLP) tasks. Unfortunately, state-of-the-art models are vulnerable to noisy texts. We propose a new contextual text denoising algorithm based on the ready-to-use masked language model. The proposed algorithm does not require retraining of the model and can be integrated into any NLP system without additional training on paired cleaning training data. We evaluate our method under synthetic noise and natural noise and show that the proposed algorithm can use context information to correct noise text and improve the performance of noisy inputs in several downstream tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
DenoisingLanguage ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
Contextual Text Denoising with Masked Language Model
Recently, with the help of deep learning models, significant advances have been made in different Natural Language Processing (NLP) tasks. Unfortunately, state-of-the-art models are vulnerable to noisy texts. We propose …
DenoisingLanguage ModelingLanguage ModellingmodelAdaptive Guidance for Retrieval-Augmented Masked Diffusion Models
Retrieval-Augmented Generation (RAG) improves factual grounding by incorporating external knowledge into language model generation. However, when retrieved context is noisy, unreliable, or inconsistent with the model's p…
Contextual Representation Learning beyond Masked Language Modeling
Currently, masked language modeling (e.g., BERT) is the prime choice to learn contextualized representations. Due to the pervasiveness, it naturally raises an interesting question: how do masked language models (MLMs) le…
Language ModelingLanguage ModellingMasked Language ModelingRepresentation LearningConditional BERT Contextual Augmentation
We propose a novel data augmentation method for labeled sentences called conditional BERT contextual augmentation. Data augmentation methods are often applied to prevent overfitting and improve generalization of deep neu…
Data AugmentationLanguage ModelingLanguage ModellingText ClassificationStructure Matters: Tackling the Semantic Discrepancy in Diffusion Models for Image Inpainting
Denoising diffusion probabilistic models for image inpainting aim to add the noise to the texture of image during the forward process and recover masked regions with unmasked ones of the texture via the reverse denoising…
DenoisingImage Inpainting